=== verify_all.sh started at 2026-04-04T15:40:12Z === 0Chain Comprehensive Verification Domain: test.zus.network Apps: test.vult.network / test.blimp.software / test.bolt.holdings 0box: https://0box.test.zus.network | Time: Sat Apr 4 05:40:12 PM CEST 2026 ══════════════════════════════════════════════════  Chain Health ══════════════════════════════════════════════════ [PASS] miner-1 running (restarts=0) [PASS] miner-2 running (restarts=0) [PASS] miner-3 running (restarts=0) [PASS] miner-4 running (restarts=0) [PASS] sharder-1 running (restarts=0) [PASS] sharder-2 running (restarts=0) [PASS] Sharder-1 responding (round 3769) [PASS] Sharder-2 responding (round 3769) [PASS] Chain advancing (round 3769 → 3775) [PASS] 4/4 miners responding [PASS] Kafka pipeline: lag=-3 (chain=3769, 0box=3772) ══════════════════════════════════════════════════  Backend Service Health ══════════════════════════════════════════════════ [PASS] 0dns reachable (HTTP 200) [PASS] 0box API reachable (HTTP 200) [PASS] zauth container running [PASS] zvault container running [PASS] ZS3 server reachable (HTTP 200) [PASS] Elasticsearch reachable (HTTP 200) [PASS] Gotenberg render service (port 3010) reachable [PASS] 0box block_worker → local chain (https://test1.zus.network/dns) [PASS] 0box deployment_mode=3 (auth bypass — Vult/Blimp/Bolt work without Firebase) [PASS] Crawler container running [PASS] Crawler allocation valid (expires Mon May 4 05:39:44 PM CEST 2026) [PASS] 0box DB: 4 active miners [PASS] 0box DB: 2 active sharders [PASS] 0box DB: 12 available blobbers [PASS] Kafka pipeline: 3774 snapshots (latest round: 3774) [PASS] Crawler allocation has 12 blobbers (all 12 regular blobbers) ══════════════════════════════════════════════════  CORS — Vult / Blimp / Bolt ══════════════════════════════════════════════════ [PASS] CORS zauth ← test.vult.network: origin=https://test.vult.network credentials=true [PASS] CORS zauth ← test.blimp.software: origin=https://test.blimp.software credentials=true [PASS] CORS zauth ← test.bolt.holdings: origin=https://test.bolt.holdings credentials=true [PASS] CORS zauth ← test.atlus.cloud: origin=https://test.atlus.cloud credentials=true [PASS] CORS sharder01 ← test.vult.network: origin=https://test.vult.network credentials=true [PASS] CORS sharder01 ← test.blimp.software: origin=https://test.blimp.software credentials=true [PASS] CORS sharder01 ← test.bolt.holdings: origin=https://test.bolt.holdings credentials=true [PASS] CORS sharder01 ← test.atlus.cloud: origin=https://test.atlus.cloud credentials=true [PASS] CORS miner01 ← test.vult.network: origin=https://test.vult.network credentials=true [PASS] CORS miner01 ← test.blimp.software: origin=https://test.blimp.software credentials=true [PASS] CORS miner01 ← test.bolt.holdings: origin=https://test.bolt.holdings credentials=true [PASS] CORS miner01 ← test.atlus.cloud: origin=https://test.atlus.cloud credentials=true [PASS] CORS 0box ← test.vult.network: origin=https://test.vult.network credentials=true [PASS] CORS 0box ← test.blimp.software: origin=https://test.blimp.software credentials=true [PASS] CORS 0box ← test.bolt.holdings: origin=https://test.bolt.holdings credentials=true [PASS] CORS 0box ← test.atlus.cloud: origin=https://test.atlus.cloud credentials=true [PASS] WebSocket upgrade passes 0box (HTTP 101) ══════════════════════════════════════════════════  Web App Serving (Vult / Bolt / Blimp / Explorer) ══════════════════════════════════════════════════ [PASS] vult running on port 3003 (HTTP 200) [PASS] bolt running on port 3002 (HTTP 200) [PASS] blimp running on port 3006 (HTTP 200) [PASS] explorer running on port 3001 (HTTP 200) [PASS] UI sub-pages: all 12/12 pages return 200/301 [PASS] UI button API endpoints: all 3/3 responding [PASS] vult custom domain https://test.vult.network/ reachable (HTTP 200) [PASS] bolt custom domain https://test.bolt.holdings/ reachable (HTTP 200) [PASS] blimp custom domain https://test.blimp.software/ reachable (HTTP 200) [PASS] explorer custom domain https://test.atlus.cloud/ reachable (HTTP 200) [PASS] vult: .env has no stale dev/prod URLs [PASS] bolt: .env has no stale dev/prod URLs [PASS] blimp: .env has no stale dev/prod URLs [PASS] shared: .env has no stale dev/prod URLs [PASS] Web app processes OK (4/4 ports serving, 00 Next.js procs) ══════════════════════════════════════════════════  Vult — Storage App Checks ══════════════════════════════════════════════════ [INFO] Checking Vult custom domain navigation pages (https://test.vult.network)... [PASS] Vult domain pages: all 4/4 reachable at https://test.vult.network [INFO] Faucet: pouring tokens to local.json... [PASS] Faucet: poured tokens successfully [PASS] Signup endpoint: working (HTTP 200 — user_id accepted) [PASS] Login: OTP verified (dev bypass working) [WARN] 0box /v2/freestorage: record not found (will fall back to regular allocation) [PASS] Allocation created (regular fallback): e984e37602ecd8ac... [FAIL] Allocation e984e37602ecd8ac... not found on chain (creation likely failed) — skipping upload tests [FAIL] Could not create allocation — skipping upload/download/carousel/replace checks [INFO] Second wallet flow: create wallet → fund → allocate → upload → switch... [PASS] Second wallet created (vw2_4128737) [PASS] Second wallet allocation: 2b16973651001824... [WARN] Second wallet upload: Error fetching the allocation. allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: 2b16973651001824f633f6f889aea47497488ccd1f3a68dbc6fe3afcbc3b4fe9, error: record not found [WARN] Switch wallet: second wallet files not visible: Error fetching the allocation allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: 2b16973651001824f633f6f889aea47497488ccd1f3a68dbc6fe3afcbc3b4fe9, error: record not found [PASS] No stale dev/devtest URLs in service configs ══════════════════════════════════════════════════  Blimp — File Manager Checks ══════════════════════════════════════════════════ [PASS] Blimp app serving on port 3006 [PASS] Blimp auth: 0box deployment_mode=3 (no Firebase required) [FAIL] 0box: only 0 regular blobbers — Blimp 'no provider match' [INFO] Fix: bash scripts/deploy_local.sh fix-kafka [PASS] Blimp carousel: Gotenberg container healthy (port 3010) [PASS] Blimp carousel: Gotenberg reachable from 0box container — carousel path OK [PASS] Blimp carousel: 0box /v2/convert-to-pdf endpoint registered (HTTP 400) [WARN] Faucet: https://test1.zus.network/sharder02/v1/transaction/get/confirmation?hash=5c48923180178f659fbc825d541af6a3de1e0d9060c7102740d5e4cc5117703f: [400] {"error":"unexpected end of JSON input"} [PASS] Blimp allocation created: e984e37602ecd8ac... [WARN] Blimp upload text: Error fetching the allocation. allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: e984e37602ecd8ac5201638b45bc63d73bd2416bf8b91d9deb514bd6819b16fc, error: record not found [WARN] Blimp upload PDF: Error fetching the allocation. allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: e984e37602ecd8ac5201638b45bc63d73bd2416bf8b91d9deb514bd6819b16fc, error: record not found [WARN] Blimp upload PNG: Error fetching the allocation. allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: e984e37602ecd8ac5201638b45bc63d73bd2416bf8b91d9deb514bd6819b16fc, error: record not found [WARN] Blimp: PDF download failed: Error fetching the allocation allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: e984e37602ecd8ac5201638b45bc63d73bd2416bf8b91d9deb514bd6819b16fc, error: record not found [WARN] Blimp Settings add blobber: no free blobber to add (all in use) [WARN] Blimp Settings repair: Error fetching the allocation. allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: e984e37602ecd8ac5201638b45bc63d73bd2416bf8b91d9deb514bd6819b16fc, error: record not found [WARN] Blimp upload-only mode: Error updating allocation:couldnt_find_allocation: Couldn't find the allocation required for update [WARN] Blimp freeze mode: Error updating allocation:couldnt_find_allocation: Couldn't find the allocation required for update [PASS] Blimp: .env has no stale dev/prod URLs ══════════════════════════════════════════════════  Bolt — Wallet App Checks ══════════════════════════════════════════════════ [PASS] Bolt app serving on port 3002 [PASS] Faucet SC: pour transaction works [PASS] Bolt wallet balance: Balance: 381.498326 ZCN (0.69 USD) [PASS] Bolt staking: 4 miners with non-zero stake (visible in Bolt UI) [PASS] Bolt auth: 0box deployment_mode=3 [PASS] Bolt send: wallet has balance=3814983260000 (send transaction possible) [PASS] Bolt: .env has no stale dev/prod URLs ══════════════════════════════════════════════════  Atlus — Explorer Dashboard Data ══════════════════════════════════════════════════ [PASS] Miners on chain: 4 [PASS] Miners with stake: 4/4 [PASS] Sharders with stake: 2/2 [PASS] Validators on chain: 12 (staked: 12, active: 12) [PASS] Blobbers on chain: 17 (enterprise: 5) [WARN] No challenge history — crawler may need time to generate activity [INFO] Checking blobber health endpoints... [FAIL] All blobber health checks failed [PASS] Provider rewards: 17 providers earning rewards [WARN] Latest block has 0 transactions (chain idle) [PASS] 0box graph write-price: 10 data points [PASS] 0box graph challenges: 10 data points [PASS] 0box graph token-supply: 10 data points [PASS] 0box graph txns-count: 10 data points [PASS] 0box graph used-storage: 10 data points [PASS] 0box graph total-allocations: 10 data points [PASS] Elasticsearch cluster: yellow [PASS] Elasticsearch has 9342 indexed documents (search functional) [INFO] Elasticsearch resources: CPU=1.63% MEM=984.1MiB [PASS] Atlus/Explorer app serving on port 3001 [INFO] Checking Atlus custom domain navigation pages (https://test.atlus.cloud)... [PASS] Atlus domain pages: all 7/7 reachable at https://test.atlus.cloud ══════════════════════════════════════════════════  Blobber Health ══════════════════════════════════════════════════ [PASS] 9/9 regular blobbers responding [PASS] 3/3 enterprise blobbers responding [PASS] All infrastructure blobbers health-checked within last hour [PASS] Blobber delegate wallets match local.json ══════════════════════════════════════════════════  Provider Rewards, Crawler & JWT ══════════════════════════════════════════════════ [PASS] POST /v2/jwt/token → 200 (JWT created) [PASS] PUT /v2/jwt/token → 200 (JWT refresh works — SignatureHandler bypassed) [PASS] Crawler allocation e835a145714d1277… is active (not finalized) [PASS] Crawler allocation has 12 blobbers (need 9) [PASS] All blobbers 01-09 present in crawler allocation [PASS] 11 blobber(s) have challenge rewards > 0 [WARN] 1 blobber(s) with 0 rewards: blobber12 — ensure they have data (crawler uploads) ══════════════════════════════════════════════════  VERIFICATION SUMMARY ══════════════════════════════════════════════════ Domain: test.zus.network (prefix: test) Vult: https://test.vult.network/ Blimp: https://test.blimp.software/ Bolt: https://test.bolt.holdings/ Atlus: https://test.atlus.cloud/ PASS: 107 WARN: 15 FAIL: 4 Failed checks: ✗ Allocation e984e37602ecd8ac... not found on chain (creation likely failed) — skipping upload tests ✗ Could not create allocation — skipping upload/download/carousel/replace checks ✗ 0box: only 0 regular blobbers — Blimp 'no provider match' ✗ All blobber health checks failed 4 FAILED — see above ALL fixes must go into scripts/deploy_local.sh or test code. Never make manual one-off changes on the server. === verify_all.sh finished at 2026-04-04T15:43:21Z exit=1 ===