Websocket Error at end of LXC Copy Operation

text |
14.51 KB
None
0 👍
0 👎
Raw Download Clone Embed Print View
Log below generated from the running the following command:
 lxc copy --mode push -d eth0,ipv4.address=10.12.13.14 my_container remote:my_container
 Aug 10 23:59:46 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:46+10:00" level=debug msg="Event listener server handler started" id=b84ea491-66bf-4e59-87ef-a4da1a433e32 local="10.20.20.94:11443" remote="10.10.10.24:39982"
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Matched trusted cert" fingerprint=e9e1552c4be67daebf9c3bfe0ac45db0c07c3c35bc9d0cc1765de36ed6315c77 subject="CN=akk@akk-tpx1,O=linuxcontainers.org"
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Handling API request" ip="10.10.10.24:39998" method=POST protocol=tls url=/1.0/instances username=e9e1552c4be67daebf9c3bfe0ac45db0c07c3c35bc9d0cc1765de36ed6315c77
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="API Request\n\t{\n\t\t\"architecture\": \"x86_64\",\n\t\t\"config\": {\n\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\"image.description\": \"ubuntu 16.04 LTS amd64 (release) (20170815.1)\",\n\t\t\t\"image.label\": \"release\",\n\t\t\t\"image.os\": \"ubuntu\",\n\t\t\t\"image.release\": \"xenial\",\n\t\t\t\"image.serial\": \"20170815.1\",\n\t\t\t\"image.version\": \"16.04\",\n\t\t\t\"limits.cpu\": \"3-3\",\n\t\t\t\"limits.memory\": \"4GB\",\n\t\t\t\"volatile.base_image\": \"58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08\"\n\t\t},\n\t\t\"devices\": {\n\t\t\t\"eth0\": {\n\t\t\t\t\"ipv4.address\": \"10.22.145.192\",\n\t\t\t\t\"name\": \"eth0\",\n\t\t\t\t\"nictype\": \"bridged\",\n\t\t\t\t\"parent\": \"lxdbr0\",\n\t\t\t\t\"type\": \"nic\"\n\t\t\t}\n\t\t},\n\t\t\"ephemeral\": false,\n\t\t\"profiles\": [\n\t\t\t\"default\"\n\t\t],\n\t\t\"stateful\": false,\n\t\t\"description\": \"\",\n\t\t\"name\": \"<REDACTED>\",\n\t\t\"source\": {\n\t\t\t\"type\": \"migration\",\n\t\t\t\"certificate\": \"\",\n\t\t\t\"base-image\": \"58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08\",\n\t\t\t\"mode\": \"push\",\n\t\t\t\"allow_inconsistent\": false\n\t\t},\n\t\t\"instance_type\": \"\",\n\t\t\"type\": \"container\"\n\t}" ip="10.10.10.24:39998" method=POST protocol=tls url=/1.0/instances username=e9e1552c4be67daebf9c3bfe0ac45db0c07c3c35bc9d0cc1765de36ed6315c77
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Responding to instance create"
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Instance operation lock created" action=create instance=<REDACTED> project=default reusable=false
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=info msg="Creating instance" ephemeral=false instance=<REDACTED> instanceType=container project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Adding device" device=eth0 instance=<REDACTED> instanceType=container project=default type=nic
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Adding device" device=root instance=<REDACTED> instanceType=container project=default type=disk
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=info msg="Created instance" ephemeral=false instance=<REDACTED> instanceType=container project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="New operation" class=websocket description="Creating instance" operation=d92d3876-8e66-4542-b38f-ce6c400d2e31 project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Started operation" class=websocket description="Creating instance" operation=d92d3876-8e66-4542-b38f-ce6c400d2e31 project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/d92d3876-8e66-4542-b38f-ce6c400d2e31\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"d92d3876-8e66-4542-b38f-ce6c400d2e31\",\n\t\t\t\"class\": \"websocket\",\n\t\t\t\"description\": \"Creating instance\",\n\t\t\t\"created_at\": \"2023-08-10T23:59:47.044461655+10:00\",\n\t\t\t\"updated_at\": \"2023-08-10T23:59:47.044461655+10:00\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/instances/<REDACTED>\"\n\t\t\t\t],\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/<REDACTED>\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": {\n\t\t\t\t\"control\": \"dfc1cff2437a968627b183a38d325fa889f828132a5cf5b8ab0b96d931fd97a5\",\n\t\t\t\t\"fs\": \"d458ef93984a0d09079e625309210f27c4786fe5f631d4cb8e659e8d56ec04e3\"\n\t\t\t},\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}" http_code=202
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=info msg="Waiting for migration control connection on target" clusterMoveSourceName= instance=<REDACTED> live=false project=default push=true
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Allowing untrusted GET" ip="10.10.10.24:40012" url="/1.0/operations/d92d3876-8e66-4542-b38f-ce6c400d2e31/websocket?secret=dfc1cff2437a968627b183a38d325fa889f828132a5cf5b8ab0b96d931fd97a5"
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Connecting to operation" class=websocket description="Creating instance" operation=d92d3876-8e66-4542-b38f-ce6c400d2e31 project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Connected to operation" class=websocket description="Creating instance" operation=d92d3876-8e66-4542-b38f-ce6c400d2e31 project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=info msg="Migration control connection established on target" clusterMoveSourceName= instance=<REDACTED> live=false project=default push=true
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=info msg="Migration receive starting" instance=<REDACTED> instanceType=container project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Allowing untrusted GET" ip="10.10.10.24:40018" url="/1.0/operations/d92d3876-8e66-4542-b38f-ce6c400d2e31/websocket?secret=d458ef93984a0d09079e625309210f27c4786fe5f631d4cb8e659e8d56ec04e3"
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Connecting to operation" class=websocket description="Creating instance" operation=d92d3876-8e66-4542-b38f-ce6c400d2e31 project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Connected to operation" class=websocket description="Creating instance" operation=d92d3876-8e66-4542-b38f-ce6c400d2e31 project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Waiting for migration offer from source" instance=<REDACTED> instanceType=container project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Sending migration response to source" instance=<REDACTED> instanceType=container project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Sent migration response to source" instance=<REDACTED> instanceType=container project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Migrate receive filesystem transfer started" instance=<REDACTED> instanceType=container project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="CreateInstanceFromMigration started" args="{IndexHeaderVersion:1 Name:<REDACTED> Description: Config:map[] Snapshots:[] MigrationType:{FSType:RSYNC Features:[xattrs delete compress bidirectional]} TrackProgress:true Refresh:false Live:false VolumeSize:0 ContentType: VolumeOnly:false ClusterMoveSourceName:}" driver=dir instance=<REDACTED> pool=default project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Waiting for migration index header" args="{IndexHeaderVersion:1 Name:<REDACTED> Description: Config:map[] Snapshots:[] MigrationType:{FSType:RSYNC Features:[xattrs delete compress bidirectional]} TrackProgress:true Refresh:false Live:false VolumeSize:0 ContentType: VolumeOnly:false ClusterMoveSourceName:}" driver=dir instance=<REDACTED> pool=default project=default version=1
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Migrate receive control monitor started" instance=<REDACTED> instanceType=container project=default
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Websocket: Got barrier message" address="10.10.10.24:40018"
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=info msg="Received migration index header, sending response" args="{IndexHeaderVersion:1 Name:<REDACTED> Description: Config:map[] Snapshots:[] MigrationType:{FSType:RSYNC Features:[xattrs delete compress bidirectional]} TrackProgress:true Refresh:false Live:false VolumeSize:0 ContentType: VolumeOnly:false ClusterMoveSourceName:}" driver=dir instance=<REDACTED> pool=default project=default version=1
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Websocket: Sending barrier message" address="10.10.10.24:40018"
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Sent migration index header response" args="{IndexHeaderVersion:1 Name:<REDACTED> Description: Config:map[] Snapshots:[] MigrationType:{FSType:RSYNC Features:[xattrs delete compress bidirectional]} TrackProgress:true Refresh:false Live:false VolumeSize:0 ContentType: VolumeOnly:false ClusterMoveSourceName:}" driver=dir instance=<REDACTED> pool=default project=default version=1
Aug 10 23:59:47 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:47+10:00" level=debug msg="Receiving filesystem volume started" driver=dir path=/var/snap/lxd/common/lxd/storage-pools/default/containers/<REDACTED>/ pool=default volName=<REDACTED>
 *** Line below repeated >600 times - only difference was the timestamp ***
 Aug 10 23:59:48 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-10T23:59:48+10:00" level=debug msg="Updated metadata for operation" class=websocket description="Creating instance" operation=d92d3876-8e66-4542-b38f-ce6c400d2e31 project=default
 *** Line above repeated >600 times ***
 Aug 11 00:18:09 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:18:09+10:00" level=debug msg="Websocket: Sending barrier message" address="10.10.10.24:40018"
Aug 11 00:18:10 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:18:10+10:00" level=debug msg="Migrate receive control monitor finished" instance=<REDACTED> instanceType=container project=default
Aug 11 00:18:10 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:18:10+10:00" level=debug msg="Event listener server handler stopped" listener=b84ea491-66bf-4e59-87ef-a4da1a433e32 local="10.20.20.94:11443" remote="10.10.10.24:39982"
Aug 11 00:18:10 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:18:10+10:00" level=debug msg="Receiving filesystem volume stopped" driver=dir path=/var/snap/lxd/common/lxd/storage-pools/default/containers/<REDACTED>/ pool=default volName=<REDACTED>
Aug 11 00:19:32 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:19:32+10:00" level=debug msg="CreateInstanceFromMigration finished" args="{IndexHeaderVersion:1 Name:<REDACTED> Description: Config:map[] Snapshots:[] MigrationType:{FSType:RSYNC Features:[xattrs delete compress bidirectional]} TrackProgress:true Refresh:false Live:false VolumeSize:0 ContentType: VolumeOnly:false ClusterMoveSourceName:}" driver=dir instance=<REDACTED> pool=default project=default
Aug 11 00:19:32 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:19:32+10:00" level=debug msg="Migrate receive filesystem transfer finished" instance=<REDACTED> instanceType=container project=default
Aug 11 00:19:32 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:19:32+10:00" level=debug msg="Sending migration failure response to source" err="Error reading migration control source: websocket: close 1000 (normal)" instance=<REDACTED> instanceType=container project=default
Aug 11 00:19:32 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:19:32+10:00" level=warning msg="Failed sending migration failure to source" err="websocket: close sent" instance=<REDACTED> instanceType=container project=default
Aug 11 00:19:32 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:19:32+10:00" level=info msg="Migration receive stopped" instance=<REDACTED> instanceType=container project=default
Aug 11 00:19:32 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:19:32+10:00" level=error msg="Failed migration on target" clusterMoveSourceName= err="Error reading migration control source: websocket: close 1000 (normal)" instance=<REDACTED> live=false project=default push=true
Aug 11 00:19:32 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:19:32+10:00" level=info msg="Migration channels disconnected on target" clusterMoveSourceName= instance=<REDACTED> live=false project=default push=true
Aug 11 00:19:32 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:19:32+10:00" level=debug msg="Instance operation lock finished" action=create err="Error transferring instance data: Failed migration on target: Error reading migration control source: websocket: close 1000 (normal)" instance=<REDACTED> project=default reusable=false
Aug 11 00:19:32 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:19:32+10:00" level=debug msg="Removing device" device=root instance=<REDACTED> instanceType=container project=default type=disk
Aug 11 00:19:32 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:19:32+10:00" level=debug msg="Removing device" device=eth0 instance=<REDACTED> instanceType=container project=default type=nic
Aug 11 00:19:32 lxcsrvr2 lxd.daemon[2424040]: time="2023-08-11T00:19:32+10:00" level=debug msg="Failure for operation" class=websocket description="Creating instance" err="Error transferring instance data: Failed migration on target: Error reading migration control source: websocket: close 1000 (normal)" operation=d92d3876-8e66-4542-b38f-ce6c400d2e31 project=default
Pasted

Websocket Error at end of LXC Copy Operation

Comments
