Astar zkEVM: Network Upgrade Report

Astar zkEVM
1

Incident Description

On April 6th, Astar zkEVM’s engineering team detected synchronization issues on the network through RPC alerts. Collaborating closely with Gelato and Polygon developers, our team did a thorough investigation to pinpoint the underlying cause.

The issue was traced back to a reorganization (reorg) on the Ethereum L1, which resulted in an invalid batch on our network. This is similar to a recent issue that affected Polygon zkEVM mainnet a few weeks back.

Solution implemented

To address this, on April 8th, the Astar zkEVM mainnet underwent an upgrade from Polygon CDK Fork ID 7 to 9. Resolving the issue also resulted in reorg of the network, as well as block explorer re-indexing and permission-less node resynchronization. During the resolution of this issue, the network experienced downtime for several hours.

As a result, transactions executed between April 6th and 8th may have been affected by the reorg, potentially processed in alternate blocks or not at all.

User impact

Astar zkEVM users should be aware that some transactions or actions might have been reversed due to the reorg. This means that certain actions, such as swaps, liquidity provisions, NFT trades, NFT mints, bridge transactions, etc, carried out on Astar zkEVM between April 6th and April 8th could be reversed.

For users who have used the native canonical bridge between Ethereum L1 and Astar zkEVM, all transactions have been correctly processed. However, withdrawals on Ethereum may be delayed as the proof has not yet been submitted.

For users who have used third-party bridges (LayerZero, Layerswap, or Relay Link), we are working with them to determine the impact of the reorg on bridge input and output transactions and will keep the community informed.

For users affected, provide as much detail as possible in the following form: https://forms.gle/GLWWBH7xkeLZzrY26

Summary

Astar zkEVM network is up and running properly now. We are working closely with infrastructure partners and dApps running and using permissionless nodes to resynchronize from the correct batch and display the correct data.

The network underwent maintenance for a brief period, during which the Astar L1 network (Substrated-based parachain) remained unaffected.

If you require further assistance or have been impacted by these developments, please don’t hesitate to contact us below or reach out to us on Discord.

23 Likes
2

Thank you for sharing. Do we have an ETA when the synchronization is finished? When we’ll be able to utilize the Astar zkEVM mainnet, dapps, Bridges, normally…

2 Likes
3

The current RPC is already completely synced.
If you use any dApp or Bridge since after the upgrade, you are all good.

The main issue is not our infra but our partners that need to resync their subgraphs to get the latest index. We will also share all instructions to those running a permissionless node in our morning. Building out the instructions now

4 Likes
4

HELP!
I have to issues now

  1. I unstaked vASTR on omni.ls (Astar zkEVM network) and during the maintenance last night the status of my unstake request changed to “Unstake success” but I have not received anything!!!

  2. After the maintenance I bridged ASTR token from Astar zkEVM network to Astar network using stargate.finance but again i have not received anything yet.

HAVE I LOST MY MONEY???
WHAT SHOULD I DO NOW???

If there is no urgent response or action from your side to solve these issues, I will share this horrible experience on YouTube and other social media.

5

Check your Astar zkEVM wallet address on the Explorer and see if you have your astr balance on the zkEVM mainnet

6

Thanks for sharing your concerns. Let me share all the answers:

  • vASTR issue: please wait until Omni.ls has synced their nodes to the new upgraded version, the moment this is done, you can proceed again. Nothing is lost but you must do the unstaking action again the moment Omni.ls is synced up.

  • Stargate issue: Stargate is using LayerZero, LayerZero currently has a backlog of transactions that are stuck. We are now working with LayerZero to remove the first failed transaction so all other transaction can be processed. Your transaction will be part of this.

TLDR:
Nothing is lost on your end, you just need to wait until all partners are fully synced back.

7

Does this mean that the lp I added at arthswap will be rolled back and won’t get any rewards for two days of repair? Does this mean I have to buy back my position at a higher price?.. btw, Now I haven’t seen any of my assets come back.

And if so, does that mean that it might happen in the future that I lose all my profits in a certain period of time.

1 Like
8

The changes made by the Polygon CDK Fork IDs 7 to 9 have me thinking if this will happen again whenever the network gets updated, or if this time was unique because of issues with syncing.

Also, the team said the network update would be on April 10, 2024, at 12 PM UTC. But, the update came earlier than expected, and they only mentioned maintenance was happening. It would be better in the future to communicate updates more clearly, like advising people to pause transactions for a bit.

I made about 500 transactions with Yoki Origins and sold an NFT for around $300 to $500, which are now gone because of the reorganization. Many people in the Discord think they’ll get their money back, but that won’t happen. I hope there will be good communication to help them understand what’s going on.

Lastly, I hope there will be an announcement for the reopening date of the Yoki Origins website instead of just opening it suddenly. The QuickSwap YoPort 50000 transaction was reversed, which greatly affects the game balance, so I want to be prepared for the reopening.

2 Likes
9

I’ve read through the forum post and announcements and could use some help understanding what happens to transactions that were carried out between the 6th and 8th that were not reversed (funds not back in original accounts), did not reach the destination, and txns are now missing from the block explorer?

Personally I have some ASTR that this has happened to but the bigger issue for me is that the HAHA team opened airdrop claims on the 6th and many of our members are commenting on the same issue: tokens gone, txns disappeared, and we are not seeing the coins back into the original account prior to airdrop claim.

2 Likes
10

I see references to trade rollback, does this mean I need to buy back my position at a higher price. By the way, about 17 hours ago I saw someone performing a purchase operation to get a cheaper token.
I don’t know if we will retain the profits from the 7-9 period, if not, it means that we have actually lost some money.
if very time this happens, the gains have to be rolled back, which is very insecure.

1 Like
11

Thank you for the detailed status report and coordination.
I also express my respect for your sincere response and efforts.

With that said, I would like to speak on behalf of others who may be concerned.
Is it possible for incidents like this to occur again in Astar zkEVM in the future?
While I believe it is impossible to completely avoid reorgs on Ethereum, we would like to know about the potential impact on Astar zkEVM, which is an L2 solution, including the frequency and extent of such events.

3 Likes
12

How can I recover the funds through the relay bridge? I think you were outrageously wrong this time and handled it quite poorly
Or do I not understand your plan?

bridge TX: 0x36635f26c80f29c4cbdde4cd47e6c81b30d756a92e05ccec3038508c35253d21

1 Like
13

Does this mean that deals from 7th to 9th will be cancelled? That was a bad experience, and now we need to pay a higher price for the assets we already bought. If such a thing happens again, does it mean that we will lose the assets we have already taken again?

15

Have some faith in the team all, this is still experimental technology and issues are bound to happen. EVERYONE forgets that from time to time.

The Astar team will make this right, just give them some time.

3 Likes
16

Thanks for sharing the info.
I will add to @tksarah 's question as I have questions on more technical details.

  1. More detailed cause of invalid batch
  2. Why did the reorg occur over a period as long as two days?

I understand that this is a very difficult task that requires a wide range of responses. However, I think this is an important issue for safe use, and I look forward to your response.

2 Likes
17

I lost ETH during this accident, how I can get it back?

18

Will the Yoki Origins campaign have any impact? when the Website is currently inaccessible and transactions from April 6-8 are reversed and reorganized.

19

wait, is the yoki origins tx between 6 and 8 positif to be reversed? i thought the wording from marteen is “might have been reversed” and “could be reversed”

20

Thank you for your transparency. I believe that Astar team is working hard to make this right!

2 Likes
21

The update itself has nothing to do with the rollback. This incident was caused by a problematic batch related to the L1 reorg. The upgrade for ForkID 9 is to ensure that future reorg issues won’t impact our L2.

We also raised our voice to Polygon that downtime for future upgrades is not acceptable. They promised that future upgrade will be done without any downtime or at least a couple of minutes to reboot the sequencer. 4-7 hours is not acceptable for an upgrade when you have a live DeFi ecosystem.

I agree 100%. The update was indeed planned for April 10th, but because of this problematic batch, we had no option to fast-track this upgrade. It was moving so fast that we didn’t had time to inform everyone in the way we want.

I’ll ensure that the post-mortem and analysis of the whole incident will be shared publicly, as well we plan a community call with all our partners (Gelato & Polygon) to answer any questions from the community. More information will be shared.

4 Likes