Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic in [email protected]/hamt/hamt.go #9063

Closed
lidel opened this issue Jun 27, 2022 · 19 comments · Fixed by #9402
Closed

panic in [email protected]/hamt/hamt.go #9063

lidel opened this issue Jun 27, 2022 · 19 comments · Fixed by #9402
Assignees
@ajnavarro
Labels
kind/bug A bug in existing code (including security flaws) need/analysis Needs further analysis before proceeding need/triage Needs initial labeling and prioritization P2 Medium: Good to have, but can wait until someone steps up topic/MFS Topic MFS topic/sharding Topic about Sharding (HAMT etc)

Comments

@lidel
Copy link
Member

lidel commented Jun 27, 2022 •
edited
Loading

Version

0.12.2

(updated to 0.13.0, now waiting to see if it reproduces there too)

Description

Extracted from #8694 (comment)

Running ipfs files mkdir -p followed by ipfs files write -c in a tight loop sometimes produces a panic:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x18688c6]

goroutine 441888916 [running]:
github.com/ipfs/go-unixfs/hamt.(*Shard).childLinkType(0xc04e3ff420, 0x0, 0x40, 0x40, 0x41)
	github.com/ipfs/[email protected]/hamt/hamt.go:293 +0x26
github.com/ipfs/go-unixfs/hamt.(*Shard).walkChildren(0xc04e3ff420, 0xc037be7880, 0x0, 0x1, 0x1)
	github.com/ipfs/[email protected]/hamt/hamt.go:400 +0x27d
github.com/ipfs/go-unixfs/hamt.parallelShardWalk.func1(0x1, 0xc03f03fc00)
	github.com/ipfs/[email protected]/hamt/hamt.go:467 +0x107
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc014813b30, 0xc0ee853f20)
	golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

Relevant code calling /api/v0/files/mkdir and /api/v0/files/write over HTTP RPC API client is here.
The panic happens randomly for them, every 2-3 days.

Asked them to update to 0.13.0 and report if it happens again.

cc @schomatis
@lidel lidel added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization P1 High: Likely tackled by core team if no one steps up need/analysis Needs further analysis before proceeding topic/MFS Topic MFS topic/sharding Topic about Sharding (HAMT etc) labels Jun 27, 2022
@lidel lidel moved this to 🥞 Todo in IPFS Shipyard Team Jun 27, 2022
@lidel lidel mentioned this issue Jun 27, 2022
Closed
3 tasks
@schomatis schomatis self-assigned this Jun 28, 2022
@schomatis
Copy link
Contributor

This is on my radar but will wait for confirmation on the new version and any possible reproduction of the issue.

@kuzdogan
Copy link

kuzdogan commented Jul 15, 2022 •
edited
Loading

Although it seemed to be stable for longer time than usual, we encountered the same issue today. Here's the output below. I added some comments starting with #. The relevant code is fully open-sourced and can be found here.

It is basically a Docker container with ipfs. Initiall it runs the entrypoint.sh which adds the whole repository of verified contracts on Sourcify, copies it to /contracts in MFS, and publishes the MFS root under this IPNS. For each new verified contract added to the repository, the files are added to their respective directory in MFS over the IPFS API here. Also, the IPNS is updated every 6 hours with a cron job with publish.sh

Here's the output of the container:

sourcify@sourcify:~$ docker logs ipfs-stable
generating ED25519 keypair...done
peer identity: 12D3KooWPT7etYMSLhG1YLjWHdQqdvEKdP8ku4y4WyHmMpoUBaTq
initializing IPFS node at /root/.ipfs
to get started, enter:

	ipfs cat /ipfs/QmQPeNsJPyVWPFDVHb77w8G42Fvo15z4bG2X8D2GhfbSXc/readme

2022-07-11T15:49:39.153Z	ERROR	provider.queue	queue/queue.go:124	Failed to enqueue cid: datastore closed
k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p
# Sleeping to let daemon initialize. Otherwise can't start adding.
Sleeping 30 seconds
Initializing daemon...
go-ipfs version: 0.13.0
Repo version: 12
System version: amd64/linux
Golang version: go1.18.3
2022/07/11 15:49:44 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/udp/4001/quic
Swarm listening on /ip4/172.18.0.7/tcp/4001
Swarm listening on /ip4/172.18.0.7/udp/4001/quic
Swarm listening on /p2p-circuit
Swarm announcing /ip4/10.10.42.2/tcp/4003
Swarm announcing /ip4/10.10.42.2/udp/4003/quic
Swarm announcing /ip4/178.19.221.38/tcp/4003
Swarm announcing /ip4/178.19.221.38/udp/4003/quic
API server listening on /ip4/0.0.0.0/tcp/5001
WebUI: http://0.0.0.0:5001/webui
Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
Daemon is ready
Sleeped 30 seconds
Mon Jul 11 15:50:13 UTC 2022
# Add takes a long time. Start time above. See below the finish time
Starting ipfs add
2022-07-11T15:53:40.036Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-11T17:08:04.920Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.720Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.773Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.774Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.896Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.898Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.899Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:08:39.323Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:09:14.892Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:09:14.895Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.078Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.079Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.079Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.079Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.079Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.542Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.542Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.542Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.611Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.611Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:12:16.557Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-11T17:12:16.557Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-11T17:12:25.387Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:12:25.390Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:12:32.087Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:12:32.087Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:13:34.328Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:14:08.773Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:14:16.242Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:14:16.242Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:14:16.243Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:14:25.599Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:15:11.617Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:15:11.617Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:15:11.618Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:15:11.618Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:15:11.619Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:15:36.299Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:18:51.774Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:18:51.774Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-11T17:20:22.176Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-11T17:20:23.173Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-11T17:20:23.320Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-11T17:20:42.076Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
Finished ipfs add! New ipfs hash: QmXYqk3Uc93xLrBtCqMixiNgeUT1gYS8eeQxVHPwuNDFF9
# Add finish time below. Took around 10 hours.
Tue Jul 12 01:51:43 UTC 2022
# Updating the old /contracts hash with the newly added one
Removing /contracts from MFSc
Removed /contracts from MFSc
Copying QmXYqk3Uc93xLrBtCqMixiNgeUT1gYS8eeQxVHPwuNDFF9 to MFS at /contracts
Copied QmXYqk3Uc93xLrBtCqMixiNgeUT1gYS8eeQxVHPwuNDFF9 to MFS at /contracts
Tue Jul 12 01:51:45 UTC 2022
Adding manifest and stats
Added manifest: QmNt8QffJBHpXpYinMScaZEwtDsiZr7bvz8iprhuCMUxbq and stats: Qma8Q9R7q3YAxyQzMX5t6yXaf7NLChcjudtLhw4omnZ7dE
Publishing rootHash QmTc7LAHwiwFZN6vVKAbN127SVv1dnt8rEGJtN25BvLase under ipns key
2022-07-12T01:51:58.792Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmTc7LAHwiwFZN6vVKAbN127SVv1dnt8rEGJtN25BvLase
Published rootHash QmTc7LAHwiwFZN6vVKAbN127SVv1dnt8rEGJtN25BvLase under ipns key
Tue Jul 12 01:52:02 UTC 2022
Successfully added and published the repository
2022-07-12T06:59:01.779Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-12T06:59:01.779Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-12T06:59:01.779Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-12T06:59:01.780Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-12T06:59:02.516Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-12T06:59:02.517Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-12T06:59:02.517Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-12T06:59:02.517Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-12T06:59:02.517Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-12T06:59:02.558Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-12T06:59:02.559Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-12T23:05:54.800Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.038Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.172Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.173Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.173Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.256Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.362Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.618Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.619Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.652Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.658Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.676Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.677Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.686Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.687Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.687Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.704Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.725Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.725Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.731Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.731Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.731Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.732Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.737Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.737Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.738Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.743Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.743Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.749Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.749Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.749Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.891Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.891Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.892Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.892Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.907Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:56.751Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:56.751Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:56.976Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:57.603Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:57.610Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:05:59.463Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-12T23:10:01.579Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-12T23:10:01.610Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-12T23:10:38.139Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-12T23:10:38.140Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-12T23:11:40.279Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-13T18:01:58.354Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-13T18:01:58.354Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-13T18:01:58.355Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-13T18:01:58.355Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-13T18:01:58.355Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-13T18:01:58.355Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-13T18:01:58.355Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-13T18:01:58.355Z	ERROR	fullrtdht	fullrt/dht.go:431	key not found in map
2022-07-14T20:02:35.241Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-14T20:02:36.642Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-14T20:07:46.990Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-14T20:08:58.739Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-14T20:08:58.740Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-14T20:09:45.324Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-14T20:09:45.325Z	ERROR	autorelay	autorelay/autorelay.go:104	failed to start relay finder
2022-07-14T20:09:49.192Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-14T20:09:49.484Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-14T20:11:38.343Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
2022-07-14T20:11:38.784Z	ERROR	dht	[email protected]/dht_net.go:47	ignoring incoming dht message while not in server mode
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x19e3bf8]

goroutine 17608019870 [running]:
github.com/ipfs/go-unixfs/hamt.(*Shard).childLinkType(0x23d59a0?, 0xc0f2a5b800?)
	github.com/ipfs/[email protected]/hamt/hamt.go:293 +0x18
github.com/ipfs/go-unixfs/hamt.(*Shard).walkChildren(0xc0a43cd730, 0xc074e53700)
	github.com/ipfs/[email protected]/hamt/hamt.go:400 +0x20c
github.com/ipfs/go-unixfs/hamt.parallelShardWalk.func1()
	github.com/ipfs/[email protected]/hamt/hamt.go:467 +0x25a
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x8d
Terminated

# Here the container starts automatically after crashing
Sleeping 30 seconds
Initializing daemon...
go-ipfs version: 0.13.0
Repo version: 12
System version: amd64/linux
Golang version: go1.18.3
Sleeped 30 seconds
Fri Jul 15 02:26:24 UTC 2022
Starting ipfs add
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Finished ipfs add! New ipfs hash:
Fri Jul 15 02:26:25 UTC 2022
Removing /contracts from MFSc
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Removed /contracts from MFSc
Copying  to MFS at /contracts
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Copied  to MFS at /contracts
Fri Jul 15 02:26:27 UTC 2022
Adding manifest and stats
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Error: lock /root/.ipfs/repo.lock: someone else has the lock
2022/07/15 02:26:29 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Added manifest:  and stats:
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/udp/4001/quic
Swarm listening on /ip4/172.18.0.7/tcp/4001
Swarm listening on /ip4/172.18.0.7/udp/4001/quic
Swarm listening on /p2p-circuit
Swarm announcing /ip4/10.10.42.2/tcp/4003
Swarm announcing /ip4/10.10.42.2/udp/4003/quic
Swarm announcing /ip4/178.19.221.38/tcp/4003
Swarm announcing /ip4/178.19.221.38/udp/4003/quic
API server listening on /ip4/0.0.0.0/tcp/5001
WebUI: http://0.0.0.0:5001/webui
Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
Daemon is ready
Publishing rootHash QmQKdndr4NGrnoBcuHkTRkLjqqieTbzhEMxZmBMF7WFNYZ under ipns key
2022-07-15T02:26:30.964Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Error: failed to complete put
Published rootHash QmQKdndr4NGrnoBcuHkTRkLjqqieTbzhEMxZmBMF7WFNYZ under ipns key
Fri Jul 15 02:26:31 UTC 2022
Successfully added and published the repository
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x19e3bf8]

goroutine 224787303 [running]:
github.com/ipfs/go-unixfs/hamt.(*Shard).childLinkType(0xc04dc29490?, 0xc025cdb8d8?)
	github.com/ipfs/[email protected]/hamt/hamt.go:293 +0x18
github.com/ipfs/go-unixfs/hamt.(*Shard).walkChildren(0xc04dc29490, 0xc05bdf39c0)
	github.com/ipfs/[email protected]/hamt/hamt.go:400 +0x20c
github.com/ipfs/go-unixfs/hamt.parallelShardWalk.func1()
	github.com/ipfs/[email protected]/hamt/hamt.go:467 +0x25a
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x8d
Terminated
# Auto-restart container after crash again
Sleeping 30 seconds
Initializing daemon...
go-ipfs version: 0.13.0
Repo version: 12
System version: amd64/linux
Golang version: go1.18.3
2022/07/15 09:27:44 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/udp/4001/quic
Swarm listening on /ip4/172.18.0.7/tcp/4001
Swarm listening on /ip4/172.18.0.7/udp/4001/quic
Swarm listening on /p2p-circuit
Swarm announcing /ip4/10.10.42.2/tcp/4003
Swarm announcing /ip4/10.10.42.2/udp/4003/quic
Swarm announcing /ip4/178.19.221.38/tcp/4003
Swarm announcing /ip4/178.19.221.38/udp/4003/quic
API server listening on /ip4/0.0.0.0/tcp/5001
WebUI: http://0.0.0.0:5001/webui
Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
Daemon is ready
Sleeped 30 seconds
Fri Jul 15 09:27:46 UTC 2022
Starting ipfs add

Also adding the output of the periodic publish operations:

root@e7ebab206713:/app# cat /var/log/cron.log
Tue Jul 12 06:00:01 UTC 2022
# Running a find script to publish the verified contract stats. This gets published in stats.json
Started find in repository for stats
# ignore these as we take the superset of `full_match` folder and not all of them are found in partial_match
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats 
Tue Jul 12 06:00:47 UTC 2022
Publishing rootHash QmZKpmqNsUFBJ4tErBfL7wanvwtRiKTWeo1J3isgMxCK6o under ipns key
2022-07-12T06:14:15.966Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmZKpmqNsUFBJ4tErBfL7wanvwtRiKTWeo1J3isgMxCK6o
Published rootHash QmZKpmqNsUFBJ4tErBfL7wanvwtRiKTWeo1J3isgMxCK6o under ipns key
Tue Jul 12 06:14:16 UTC 2022
Tue Jul 12 12:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Tue Jul 12 12:00:42 UTC 2022
Publishing rootHash QmRV5j64E5xJV5RxXScuiakcc9n9o1prxjxNWByuzpyixY under ipns key
2022-07-12T12:25:18.670Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmRV5j64E5xJV5RxXScuiakcc9n9o1prxjxNWByuzpyixY
Published rootHash QmRV5j64E5xJV5RxXScuiakcc9n9o1prxjxNWByuzpyixY under ipns key
Tue Jul 12 12:25:20 UTC 2022
Tue Jul 12 18:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Tue Jul 12 18:00:45 UTC 2022
Publishing rootHash QmV4kAkBjCWbKAkvM54wHftVhUyPnDLeLBipiueLLXpi2F under ipns key
2022-07-12T18:43:17.596Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmV4kAkBjCWbKAkvM54wHftVhUyPnDLeLBipiueLLXpi2F
Published rootHash QmV4kAkBjCWbKAkvM54wHftVhUyPnDLeLBipiueLLXpi2F under ipns key
Tue Jul 12 18:43:18 UTC 2022
Wed Jul 13 00:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Wed Jul 13 00:00:42 UTC 2022
Publishing rootHash QmTt4Sh8HjzRie4wDCjVES2qqxKtEEWv5VzezXWATLtZ2z under ipns key
2022-07-13T00:41:56.020Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmTt4Sh8HjzRie4wDCjVES2qqxKtEEWv5VzezXWATLtZ2z
Published rootHash QmTt4Sh8HjzRie4wDCjVES2qqxKtEEWv5VzezXWATLtZ2z under ipns key
Wed Jul 13 00:41:58 UTC 2022
Wed Jul 13 06:00:05 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Wed Jul 13 06:00:46 UTC 2022
Publishing rootHash QmecqaAh9boMrrAXdPhuEtDrra1GdTrYGiDRsAkfczzYzk under ipns key
2022-07-13T06:51:54.754Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmecqaAh9boMrrAXdPhuEtDrra1GdTrYGiDRsAkfczzYzk
Published rootHash QmecqaAh9boMrrAXdPhuEtDrra1GdTrYGiDRsAkfczzYzk under ipns key
Wed Jul 13 06:51:59 UTC 2022
Wed Jul 13 12:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Wed Jul 13 12:00:49 UTC 2022
Publishing rootHash QmNXhrdk9p7VZN3YXLDtvgN8bXGpAbQLgfG9wMgb4yV4wA under ipns key
2022-07-13T13:31:12.516Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmNXhrdk9p7VZN3YXLDtvgN8bXGpAbQLgfG9wMgb4yV4wA
Published rootHash QmNXhrdk9p7VZN3YXLDtvgN8bXGpAbQLgfG9wMgb4yV4wA under ipns key
Wed Jul 13 13:31:13 UTC 2022
Wed Jul 13 18:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Wed Jul 13 18:00:43 UTC 2022
Publishing rootHash QmQjBWuVyRvL3jqwVGd9Jb4931hHdFGm1xDr4kGGuov3NP under ipns key
2022-07-13T19:38:46.913Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmQjBWuVyRvL3jqwVGd9Jb4931hHdFGm1xDr4kGGuov3NP
Published rootHash QmQjBWuVyRvL3jqwVGd9Jb4931hHdFGm1xDr4kGGuov3NP under ipns key
Wed Jul 13 19:38:47 UTC 2022
Thu Jul 14 00:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Thu Jul 14 00:00:48 UTC 2022
Publishing rootHash QmUGJR8aHNfQ5wAUKTcZrawcWcz3X5CECm4jy9422PQr9e under ipns key
2022-07-14T01:21:09.421Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmUGJR8aHNfQ5wAUKTcZrawcWcz3X5CECm4jy9422PQr9e
Published rootHash QmUGJR8aHNfQ5wAUKTcZrawcWcz3X5CECm4jy9422PQr9e under ipns key
Thu Jul 14 01:21:10 UTC 2022
Thu Jul 14 06:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Thu Jul 14 06:00:46 UTC 2022
Publishing rootHash QmbWG3hiccqcaWjUpNuTBvoTCHJBeYTyaAMGdoqdY6dArz under ipns key
2022-07-14T08:08:09.758Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmbWG3hiccqcaWjUpNuTBvoTCHJBeYTyaAMGdoqdY6dArz
Published rootHash QmbWG3hiccqcaWjUpNuTBvoTCHJBeYTyaAMGdoqdY6dArz under ipns key
Thu Jul 14 08:08:11 UTC 2022
Thu Jul 14 12:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Thu Jul 14 12:00:52 UTC 2022
Publishing rootHash QmNou1yXp1mJ7rViyzbxzabLe3vesDqPjgBVy4L5mrT6nb under ipns key
2022-07-14T14:17:03.134Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmNou1yXp1mJ7rViyzbxzabLe3vesDqPjgBVy4L5mrT6nb
Published rootHash QmNou1yXp1mJ7rViyzbxzabLe3vesDqPjgBVy4L5mrT6nb under ipns key
Thu Jul 14 14:17:04 UTC 2022
Thu Jul 14 18:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Thu Jul 14 18:00:45 UTC 2022
Publishing rootHash QmZvg25sMyDXpimS4EvhY78jNn4Tjr7cjYdhZL4S4zULmN under ipns key
2022-07-14T20:48:37.973Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmZvg25sMyDXpimS4EvhY78jNn4Tjr7cjYdhZL4S4zULmN
Published rootHash QmZvg25sMyDXpimS4EvhY78jNn4Tjr7cjYdhZL4S4zULmN under ipns key
Thu Jul 14 20:48:39 UTC 2022
Fri Jul 15 00:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Fri Jul 15 00:00:46 UTC 2022
Publishing rootHash QmTQyH1DkdhuEHr4XeyCjQeSv1cF42AEZ9uogqWp1sZLsC under ipns key
2022-07-15T02:03:56.096Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmTQyH1DkdhuEHr4XeyCjQeSv1cF42AEZ9uogqWp1sZLsC
Published rootHash QmTQyH1DkdhuEHr4XeyCjQeSv1cF42AEZ9uogqWp1sZLsC under ipns key
Fri Jul 15 02:03:57 UTC 2022
Fri Jul 15 06:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Fri Jul 15 06:00:45 UTC 2022
Publishing rootHash QmYqWuNj4WaGgNsoxo4ff7W8MzKLnag7kmozR6EvJXT4Mo under ipns key
2022-07-15T06:07:11.654Z	DEBUG	cmd/ipfs	ipfs/main.go:140	config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmYqWuNj4WaGgNsoxo4ff7W8MzKLnag7kmozR6EvJXT4Mo
Published rootHash QmYqWuNj4WaGgNsoxo4ff7W8MzKLnag7kmozR6EvJXT4Mo under ipns key
Fri Jul 15 06:07:15 UTC 2022

One additional thing to note is that the ipfs node is incredibly resource intensive in our case. Our server nearly always run at full capacity with ~350/400% of the CPU and around 8GB of RAM consumed by the ipfs node.

@schomatis
Copy link
Contributor

Thanks for the report @kuzdogan; will take a closer look next week but seems consistent with the original report panicking in our new custom parallel walk.

@schomatis schomatis moved this from 🥞 Todo to 🏃‍♀️ In Progress in IPFS Shipyard Team Jul 19, 2022
@schomatis schomatis mentioned this issue Jul 20, 2022
Closed
@schomatis
Copy link
Contributor

Avoiding panic on ipfs/go-unixfs#123 (blocking this on landing it). Other than that I have no idea what to do (or how to investigate this further) other than letting it roll and see if we can get more information through the added log line.

@schomatis schomatis moved this from 🏃‍♀️ In Progress to 🛑 Blocked in IPFS Shipyard Team Jul 20, 2022
@ajnavarro
Copy link
Member

Fixing the problem here: ipfs/go-unixfs#124

@ajnavarro ajnavarro moved this from 🛑 Blocked to 🔎 In Review in IPFS Shipyard Team Jul 25, 2022
@schomatis
Copy link
Contributor

@ajnavarro Could you point me to where did we have the race in our parallel walk?

@ajnavarro
Copy link
Member

Any call to these Shard methods (Set, SetLink, Swap, Remove, Take) on different threads can cause a race condition on both cases, reading or writing, and corrupt slice data.

@schomatis
Copy link
Contributor

schomatis commented Jul 26, 2022 •
edited
Loading

@ajnavarro Yes, but where are those threads created that manipulate the same shard? By that criteria anything that doesn't have a lock has a race condition, but it depends on how its being used. I need more specifics to understand why is this panic specifically being fixed by ipfs/go-unixfs#124.

@ajnavarro
Copy link
Member

ajnavarro commented Jul 28, 2022 •
edited
Loading

@kuzdogan, @schomatis I tried to reproduce the error locally with no luck.

I tried with publish.sh and entrypoint.sh to reproduce the error.

One thing I noticed is the add process takes only 10-15 mins for me instead of 10h (add 13Gb into kubo):

Add finish time below. Took around 10 hours.

Are you using mechanical disks? I try to discard if I'm doing something incorrectly.

@ajnavarro ajnavarro added need/author-input Needs input from the original author and removed need/author-input Needs input from the original author labels Aug 4, 2022
@BigLep
Copy link
Contributor

BigLep commented Aug 4, 2022

2022-08-04 maintainer conversation: hypothesis is that MFS is related, that it's potentially not following certain concurrency expectations.

Things we can do: fuzzing at the top level (run a hundred time with large sharded directories).

@Jorropo is going to paste in the link for a relevant Discord conversation.

@Jorropo
Copy link
Contributor

Jorropo commented Aug 4, 2022

https://discord.com/channels/806902334369824788/847893063841349652/1003259317383868426

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1a37eb8]

goroutine 576316133 [running]:
github.com/ipfs/go-unixfs/hamt.(Shard).childLinkType(0xc037cfc690?, 0xc00fc1e8f0?)
        github.com/ipfs/[email protected]/hamt/hamt.go:313 +0x18
github.com/ipfs/go-unixfs/hamt.(Shard).walkChildren(0xc037cfc690, 0xc0f335f380)
        github.com/ipfs/[email protected]/hamt/hamt.go:420 +0x20c
github.com/ipfs/go-unixfs/hamt.parallelShardWalk.func1()
        github.com/ipfs/[email protected]/hamt/hamt.go:487 +0x25a
golang.org/x/sync/errgroup.(Group).Go.func1()
        golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(Group).Go
        golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x8d

@BigLep BigLep moved this from 🔎 In Review to 🏃‍♀️ In Progress in IPFS Shipyard Team Aug 4, 2022
@ajnavarro
Copy link
Member

Some more context from @aschmahmann on #ipfs-implementers channel:

aschmahmann: I have a suspicion that there were some assumptions here in the past that various functions for go-unixfs were not used concurrently that is not being upheld in go-mfs. I don't know if there are other errors that would've come in a similar form if sharded directories has been enabled by default in the past, or if this error is unique due to adding some concurrency in sharded directory enumeration that came with enabling by default.
stebalien any insight on to what the historical concurrency boundaries were supposed to be?
It seems like we either:
1. Need more docs in go-unixfs explaining it's not thread safe, and then add thread-safety into go-mfs
2. Need to add locking in go-unixfs to enforce thread-safety and then document that functionality of go-unixfs

stebalien: There is no safe way to use these concurrently at the moment.

aschmahmann: So then likely the above panic is due to mfs lack of concurrency controls rather than our introduction of parallelWalk (https://github.com/ipfs/go-unixfs/blob/4571136c15d0678006f09ebde615d0c45c126ff9/hamt/hamt.go#L468) itself being problematic?

@ajnavarro ajnavarro mentioned this issue Aug 5, 2022
Closed
@schomatis schomatis removed their assignment Aug 15, 2022
@BigLep
Copy link
Contributor

BigLep commented Sep 1, 2022

What's the next step here?

Given we've had this open for weeks, do we believe this is a P1?

@ajnavarro
Copy link
Member

The following steps should be to do some Fuzz testing on mfs and check if we can reproduce the error.

Fixing attempt here: ipfs/go-unixfs#124, but maybe we want to do the blocking at mfs level.

@BigLep BigLep added P2 Medium: Good to have, but can wait until someone steps up and removed P1 High: Likely tackled by core team if no one steps up labels Sep 1, 2022
@BigLep
Copy link
Contributor

BigLep commented Sep 1, 2022

Lowering the priority because:

  1. We expect users won't hit this once feat(cmds/add): --to-files option as files cp #8927 lands.
  2. We don't have an easy repro case

When we pick this up again, we need to:

  1. Get repro (e.g., using fuzzing as suggested above)
  2. Added locking at the MFS level rather than go-unixfs.

@BigLep BigLep moved this from 🏃‍♀️ In Progress to 🥞 Todo in IPFS Shipyard Team Sep 1, 2022
@BigLep
Copy link
Contributor

BigLep commented Sep 8, 2022

2022-09-08 conversation:
@ajnavarro on 2022-09-09 will:

  1. Get feat(cmds/add): --to-files option as files cp #8927 landed
  2. Attempt to reproduce this issue with fuzzing

@ajnavarro ajnavarro mentioned this issue Sep 9, 2022
Closed
@ajnavarro
Copy link
Member

I have good and bad news.

I was able to reproduce a panic on Shard from MFS here: ipfs/go-mfs#103.
The bad news is that is a panic on a different line 😞

@ajnavarro
Copy link
Member

Nil pointers are coming when childer is nil. This can only happen on https://github.com/ipfs/go-unixfs/blob/2c23c3ea6fae3ef1b487cfc0c606a4ffc7893676/hamt/hamt.go#L798 , when swapping values that don't exist, so a new value is added.

@ajnavarro ajnavarro mentioned this issue Sep 23, 2022
Merged
@ajnavarro ajnavarro moved this from 🥞 Todo to 🏃‍♀️ In Progress in IPFS Shipyard Team Sep 23, 2022
@ajnavarro ajnavarro mentioned this issue Nov 15, 2022
Merged
@lidel
Copy link
Member Author

lidel commented Nov 16, 2022 •
edited
Loading

(iiuc) this was fixed in #9402 and will be included in Kubo 0.17.0-rc2

@lidel lidel closed this as completed Nov 16, 2022
Repository owner moved this from 🏃‍♀️ In Progress to 🎉 Done in IPFS Shipyard Team Nov 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ajnavarro ajnavarro

Labels
kind/bug A bug in existing code (including security flaws) need/analysis Needs further analysis before proceeding need/triage Needs initial labeling and prioritization P2 Medium: Good to have, but can wait until someone steps up topic/MFS Topic MFS topic/sharding Topic about Sharding (HAMT etc)
Projects
No open projects
IPFS Shipyard Team
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: update go-unixfs to v0.4.1 ipfs/kubo
6 participants
@BigLep @lidel @ajnavarro @schomatis @kuzdogan @Jorropo