Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix cassandra metrics: Malformed IPv6 address at index #1844

Merged
merged 7 commits into from
Apr 25, 2022
Merged

fix cassandra metrics: Malformed IPv6 address at index #1844

merged 7 commits into from
Apr 25, 2022

Conversation

javeme
Copy link
Contributor

@javeme javeme commented Apr 24, 2022

fix #1843
Change-Id: Ida6b7f2675c86b6b241977075f0ef137a38fdd8d

@javeme
log cassandra probe host
5db9048 
Change-Id: Ida6b7f2675c86b6b241977075f0ef137a38fdd8d
@codecov
Copy link

codecov bot commented Apr 24, 2022 •
edited
Loading

Codecov Report

Merging #1844 (85821d1) into master (9a8259e) will increase coverage by 3.73%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #1844      +/-   ##
============================================
+ Coverage     66.93%   70.66%   +3.73%     
- Complexity      444      980     +536     
============================================
  Files           446      446              
  Lines         37966    37970       +4     
  Branches       5410     5411       +1     
============================================
+ Hits          25413    26832    +1419     
+ Misses         9957     8410    -1547     
- Partials       2596     2728     +132
Impacted Files Coverage Δ
...raph/backend/store/cassandra/CassandraMetrics.java 23.94% <100.00%> (+23.94%) ⬆️
...a/com/baidu/hugegraph/backend/query/Aggregate.java 82.35% <0.00%> (ø)
...hugegraph/backend/serializer/BinarySerializer.java 85.71% <0.00%> (ø)
...java/com/baidu/hugegraph/structure/HugeVertex.java 79.50% <0.00%> (+0.35%) ⬆️
.../baidu/hugegraph/backend/query/ConditionQuery.java 85.92% <0.00%> (+0.48%) ⬆️
...in/java/com/baidu/hugegraph/backend/id/EdgeId.java 78.57% <0.00%> (+0.89%) ⬆️
.../baidu/hugegraph/backend/tx/SchemaTransaction.java 90.45% <0.00%> (+1.14%) ⬆️
...va/com/baidu/hugegraph/util/collection/IntSet.java 74.89% <0.00%> (+1.27%) ⬆️
...java/com/baidu/hugegraph/type/define/DataType.java 87.12% <0.00%> (+1.98%) ⬆️
...m/baidu/hugegraph/backend/tx/GraphTransaction.java 80.06% <0.00%> (+2.24%) ⬆️
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a8259e...85821d1. Read the comment docs.

@javeme
Copy link
Contributor Author

javeme commented Apr 24, 2022 •
edited
Loading

CassandraMetrics.newNodeProbe():

    private NodeProbe newNodeProbe(String host) throws IOException {
        LOG.info("Probe to cassandra node: '{}:{}'", host,  this.port);
        return this.username.isEmpty() ?
               new NodeProbe(host, this.port) :
               new NodeProbe(host, this.port, this.username, this.password);
    }
// log output
2022-04-24 04:45:07 [grizzly-http-server-2] [INFO] c.b.h.b.s.c.CassandraMetrics - Probe to cassandra node: '127.0.0.1:7199'

JDK version:

Downloading Java 8.0.332+9 (Zulu) from https://cdn.azul.com/zulu/bin/zulu8.62.0.19-ca-jdk8.0.332-linux_x64.tar.gz ...

@javeme
log error info
c91fa0a 
Change-Id: I18314c3b713304048e1570584ab81998c66366b3
@javeme
Copy link
Contributor Author

javeme commented Apr 24, 2022 •
edited
Loading

error stack trace:

2022-04-24 07:22:40 [grizzly-http-server-2] [INFO] c.b.h.b.s.c.CassandraMetrics - Probe to cassandra node: '127.0.0.1:7199'
2022-04-24 07:22:40 [grizzly-http-server-2] [WARN] c.b.h.b.s.c.CassandraMetrics - Unable to get metrics from host '127.0.0.1':
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.InvalidNameException: Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199 [Root exception is java.lang.IllegalArgumentException: Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[?:1.8.0_332]
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[?:1.8.0_332]
	at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:191) ~[cassandra-all-3.10.jar:3.10]
	at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:158) ~[cassandra-all-3.10.jar:3.10]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.newNodeProbe(CassandraMetrics.java:308) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.getMetricsByHost(CassandraMetrics.java:100) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.executeAllHosts(CassandraMetrics.java:299) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.metrics(CassandraMetrics.java:86) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraStore.lambda$registerMetaHandlers$0(CassandraStore.java:99) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.MetaDispatcher.dispatchMetaHandler(MetaDispatcher.java:45) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.backend.store.AbstractBackendStore.metadata(AbstractBackendStore.java:53) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.backend.tx.AbstractTransaction.metadata(AbstractTransaction.java:109) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.StandardHugeGraph.metadata(StandardHugeGraph.java:975) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.metadata(HugeGraphAuthProxy.java:669) ~[hugegraph-api-0.13.0.jar:0.67.0.0]
	at com.baidu.hugegraph.api.metrics.MetricsAPI.backend(MetricsAPI.java:87) ~[hugegraph-api-0.13.0.jar:0.67.0.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_332]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_332]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_332]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) ~[jersey-container-grizzly2-http-2.25.1.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) ~[grizzly-http-server-2.4.4.jar:2.4.4]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) ~[grizzly-framework-2.4.4.jar:2.4.4]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) ~[grizzly-framework-2.4.4.jar:2.4.4]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
Caused by: javax.naming.InvalidNameException: Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.newNamingException(rmiURLContext.java:295) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parseCompat(rmiURLContext.java:223) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parse(rmiURLContext.java:109) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext.getRootURLContext(rmiURLContext.java:314) ~[?:1.8.0_332]
	at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:215) ~[?:1.8.0_332]
	at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_332]
	... 40 more
Caused by: java.lang.IllegalArgumentException: Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199
	at java.net.URI.create(URI.java:852) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parseCompat(rmiURLContext.java:213) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parse(rmiURLContext.java:109) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext.getRootURLContext(rmiURLContext.java:314) ~[?:1.8.0_332]
	at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:215) ~[?:1.8.0_332]
	at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_332]
	... 40 more
Caused by: java.net.URISyntaxException: Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199
	at java.net.URI$Parser.fail(URI.java:2873) ~[?:1.8.0_332]
	at java.net.URI$Parser.parseIPv6Reference(URI.java:3494) ~[?:1.8.0_332]
	at java.net.URI$Parser.parseServer(URI.java:3244) ~[?:1.8.0_332]
	at java.net.URI$Parser.parseAuthority(URI.java:3180) ~[?:1.8.0_332]
	at java.net.URI$Parser.parseHierarchical(URI.java:3122) ~[?:1.8.0_332]
	at java.net.URI$Parser.parse(URI.java:3078) ~[?:1.8.0_332]
	at java.net.URI.<init>(URI.java:588) ~[?:1.8.0_332]
	at java.net.URI.create(URI.java:850) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parseCompat(rmiURLContext.java:213) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parse(rmiURLContext.java:109) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext.getRootURLContext(rmiURLContext.java:314) ~[?:1.8.0_332]
	at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:215) ~[?:1.8.0_332]
	at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_332]
	... 40 more

URI source code:
https://github.com/frohoff/jdk8u-jdk/blob/da0da73ab82ed714dc5be94acd2f0d00fbdfe2e9/src/share/classes/java/net/URI.java#L3446

@javeme
Copy link
Contributor Author

javeme commented Apr 24, 2022 •
edited
Loading

we may need to translate ipv4 to ipv6 to fix this issue, because cassandra NodeProbe has assumed that it's an ipv6 formatted address:

how to translate ipv4 to ipv6:

@javeme
Translate IPv4 to IPv6 to fix issue #1843
667d843 
Change-Id: I1e04122d9e0cd7cd5986e9b33b0fad74d22c1937
@javeme
Copy link
Contributor Author

javeme commented Apr 24, 2022 •
edited
Loading

After using an IPv4-mapped IPv6 address ::127.0.0.1, there are still errors:

2022-04-24 08:18:41 [grizzly-http-server-1] [INFO] c.b.h.b.s.c.CassandraMetrics - Probe to cassandra node: '::127.0.0.1:7199'
2022-04-24 08:18:41 [grizzly-http-server-1] [WARN] c.b.h.b.s.c.CassandraMetrics - Unable to get metrics from host '::127.0.0.1':
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: ::127.0.0.1; nested exception is: 
	java.net.SocketException: Network is unreachable (connect failed)]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[?:1.8.0_332]
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[?:1.8.0_332]
	at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:191) ~[cassandra-all-3.10.jar:3.10]
	at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:158) ~[cassandra-all-3.10.jar:3.10]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.newNodeProbe(CassandraMetrics.java:320) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.getMetricsByHost(CassandraMetrics.java:103) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.executeAllHosts(CassandraMetrics.java:311) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.metrics(CassandraMetrics.java:89) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraStore.lambda$registerMetaHandlers$0(CassandraStore.java:99) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.MetaDispatcher.dispatchMetaHandler(MetaDispatcher.java:45) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.backend.store.AbstractBackendStore.metadata(AbstractBackendStore.java:53) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.backend.tx.AbstractTransaction.metadata(AbstractTransaction.java:109) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.StandardHugeGraph.metadata(StandardHugeGraph.java:975) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.metadata(HugeGraphAuthProxy.java:669) ~[hugegraph-api-0.13.0.jar:0.67.0.0]
	at com.baidu.hugegraph.api.metrics.MetricsAPI.backend(MetricsAPI.java:87) ~[hugegraph-api-0.13.0.jar:0.67.0.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_332]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_332]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_332]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) ~[jersey-container-grizzly2-http-2.25.1.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) ~[grizzly-http-server-2.4.4.jar:2.4.4]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) ~[grizzly-framework-2.4.4.jar:2.4.4]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) ~[grizzly-framework-2.4.4.jar:2.4.4]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
Caused by: javax.naming.CommunicationException
	at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136) ~[?:1.8.0_332]
	at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:218) ~[?:1.8.0_332]
	at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_332]
	... 40 more
Caused by: java.rmi.ConnectIOException: Exception creating connection to: ::127.0.0.1; nested exception is: 
	java.net.SocketException: Network is unreachable (connect failed)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:635) ~[?:1.8.0_332]
	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[?:1.8.0_332]
	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[?:1.8.0_332]
	at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343) ~[?:1.8.0_332]
	at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116) ~[?:1.8.0_332]
	at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132) ~[?:1.8.0_332]
	at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:218) ~[?:1.8.0_332]
	at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_332]
	... 40 more
Caused by: java.net.SocketException: Network is unreachable (connect failed)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_332]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_332]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_332]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_332]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_332]
	at java.net.Socket.connect(Socket.java:613) ~[?:1.8.0_332]
	at java.net.Socket.connect(Socket.java:561) ~[?:1.8.0_332]
	at java.net.Socket.<init>(Socket.java:457) ~[?:1.8.0_332]
	at java.net.Socket.<init>(Socket.java:234) ~[?:1.8.0_332]
	at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[?:1.8.0_332]
	at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148) ~[?:1.8.0_332]
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617) ~[?:1.8.0_332]
	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[?:1.8.0_332]
	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[?:1.8.0_332]
	at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343) ~[?:1.8.0_332]
	at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116) ~[?:1.8.0_332]
	at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132) ~[?:1.8.0_332]
	at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:218) ~[?:1.8.0_332]
	at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_332]
	... 40 more

Possible reasons:

https://twiki.cern.ch/twiki/pub/EGEE/IPv6FollowUp/Introduction_to_IPv6_programming_C_Java_PHP_perl.pdf

image

https://www.masterraghu.com/subjects/np/introduction/unix_network_programming_v1.3/ch12lev1sec2.html

image

javeme added 2 commits April 24, 2022 18:53
@javeme
remove '[]' from NodeProbe.fmtUrl
38f515c 
Change-Id: I1a8ef310227acfc00c023104c172779a5a511cbe
@javeme
log Patch NodeProbe.fmtUrl
5824503 
Change-Id: I20291e6a00875f4287d2ae1960693cae354f4f38
@javeme
Copy link
Contributor Author

javeme commented Apr 24, 2022 •
edited
Loading

maybe also related to cassandra server: https://github.com/apache/cassandra/blob/bf96367f4d55692017e144980cf17963e31df127/src/java/org/apache/cassandra/utils/JMXServerUtils.java#L141

After removing '[]' from NodeProbe.fmtUrl, there are still errors:

2022-04-24 12:13:58 [grizzly-http-server-3] [INFO] c.b.h.b.s.c.CassandraMetrics - Patch NodeProbe.fmtUrl='service:jmx:rmi:///jndi/rmi://%s:%d/jmxrmi'
2022-04-24 12:13:58 [grizzly-http-server-3] [INFO] c.b.h.b.s.c.CassandraMetrics - Probe to cassandra node: '127.0.0.1:7199'
2022-04-24 12:13:58 [grizzly-http-server-3] [WARN] c.b.h.b.s.c.CassandraMetrics - Unable to get metrics from host '127.0.0.1':
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.InvalidNameException: Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199 [Root exception is java.lang.IllegalArgumentException: Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[?:1.8.0_332]
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[?:1.8.0_332]
	at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:191) ~[cassandra-all-3.10.jar:3.10]
	at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:158) ~[cassandra-all-3.10.jar:3.10]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.newNodeProbe(CassandraMetrics.java:320) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.getMetricsByHost(CassandraMetrics.java:111) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.executeAllHosts(CassandraMetrics.java:311) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraMetrics.metrics(CassandraMetrics.java:97) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.cassandra.CassandraStore.lambda$registerMetaHandlers$0(CassandraStore.java:99) ~[hugegraph-cassandra-0.13.0.jar:?]
	at com.baidu.hugegraph.backend.store.MetaDispatcher.dispatchMetaHandler(MetaDispatcher.java:45) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.backend.store.AbstractBackendStore.metadata(AbstractBackendStore.java:53) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.backend.tx.AbstractTransaction.metadata(AbstractTransaction.java:109) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.StandardHugeGraph.metadata(StandardHugeGraph.java:975) ~[hugegraph-core-0.13.0.jar:0.13.0.0]
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.metadata(HugeGraphAuthProxy.java:669) ~[hugegraph-api-0.13.0.jar:0.67.0.0]
	at com.baidu.hugegraph.api.metrics.MetricsAPI.backend(MetricsAPI.java:87) ~[hugegraph-api-0.13.0.jar:0.67.0.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_332]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_332]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_332]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) ~[jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) ~[jersey-container-grizzly2-http-2.25.1.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) ~[grizzly-http-server-2.4.4.jar:2.4.4]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) ~[grizzly-framework-2.4.4.jar:2.4.4]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) ~[grizzly-framework-2.4.4.jar:2.4.4]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
Caused by: javax.naming.InvalidNameException: Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.newNamingException(rmiURLContext.java:295) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parseCompat(rmiURLContext.java:223) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parse(rmiURLContext.java:109) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext.getRootURLContext(rmiURLContext.java:314) ~[?:1.8.0_332]
	at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:215) ~[?:1.8.0_332]
	at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_332]
	... 40 more
Caused by: java.lang.IllegalArgumentException: Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199
	at java.net.URI.create(URI.java:852) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parseCompat(rmiURLContext.java:213) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parse(rmiURLContext.java:109) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext.getRootURLContext(rmiURLContext.java:314) ~[?:1.8.0_332]
	at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:215) ~[?:1.8.0_332]
	at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_332]
	... 40 more
Caused by: java.net.URISyntaxException: Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199
	at java.net.URI$Parser.fail(URI.java:2873) ~[?:1.8.0_332]
	at java.net.URI$Parser.parseIPv6Reference(URI.java:3494) ~[?:1.8.0_332]
	at java.net.URI$Parser.parseServer(URI.java:3244) ~[?:1.8.0_332]
	at java.net.URI$Parser.parseAuthority(URI.java:3180) ~[?:1.8.0_332]
	at java.net.URI$Parser.parseHierarchical(URI.java:3122) ~[?:1.8.0_332]
	at java.net.URI$Parser.parse(URI.java:3078) ~[?:1.8.0_332]
	at java.net.URI.<init>(URI.java:588) ~[?:1.8.0_332]
	at java.net.URI.create(URI.java:850) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parseCompat(rmiURLContext.java:213) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext$Parser.parse(rmiURLContext.java:109) ~[?:1.8.0_332]
	at com.sun.jndi.url.rmi.rmiURLContext.getRootURLContext(rmiURLContext.java:314) ~[?:1.8.0_332]
	at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:215) ~[?:1.8.0_332]
	at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:1.8.0_332]
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_332]
	... 40 more

javeme added 2 commits April 24, 2022 23:53
@javeme
revert setFinalInternalState() since ineffective
2df364c 
Change-Id: I5627e9af13f89e0593ffed320f213c8b5b330735
@javeme
disable cassandra/scylladb metrics test case as workaround
85821d1 
Change-Id: I35b6f99468cfbb1ecc0dc066fbc8f1f95498a3da
@javeme
Copy link
Contributor Author

javeme commented Apr 24, 2022

We are unable to solve this problem currently, temporarily disable the testMetricsBackend() of Cassandra/ScyllaDB as a workaround.

@zyxxoo zyxxoo merged commit e6a24fc into master Apr 25, 2022
@javeme javeme deleted the cassandra-metrics-ip6-fix branch April 25, 2022 06:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zyxxoo zyxxoo zyxxoo approved these changes

@seagle-yuan seagle-yuan seagle-yuan approved these changes

@imbajin imbajin imbajin approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] Failed to retrieve RMIServer stub: Malformed IPv6 address at index 7
4 participants
@javeme @zyxxoo @seagle-yuan @imbajin