Troubleshooting

Here is a collection of useful commands and tools to troubleshoot VOLTHA. The project is still in a very early phase, so this section of the guide is more focus on how to collection information and packet dumps than it is on operations.

Depending on your issue useful tips and tricks can be found: - No ONUs are discovered - Can’t authenticate - DHCP is not working - No Ping

Trace Packets across components

Given that VOLTHA is a collection of microservices we strongly suggest the usage of a log aggregator like stern. installed. You can then aggregate logs from multiple containers in a simple way, for example to track all the packet-ins and packet-outs:

$ stern -n voltha "voltha|adapter" | grep -E "packet-out|packet-in" > packets.trace

Once you have captured the packets you need, you can see them in wireshark by transforming the logs with:

sed -n 's/.*packet":"\(.*\)",.*$/\1/p' packets.trace | sed -e 's/.\{2\}/& /g' | sed -e 's/^/000000 /' > packets.hex

And then in wireshark select File -> Import from Hex Dump

Or you can decode a single packet using this online tool: https://hpd.gasmi.net

To get all the EAPOL packets: cat packets.trace | grep 888e

or if you have hl installed you can highlight with:

cat packets.trace | grep 888e | hl -m '.*packet-in.*' -g '.*packet-out.*'

To get all the DHCP packets: cat packets.trace | grep 8100

or if you have hl installed you can highlight with:

cat packets.trace | grep 8100 | hl -m '.*packet-in.*' -g '.*packet-out.*'

OMCI Packet Dump

In order to see OMCI packets in Wireshark you’ll need to install omci.lua and BinDecHex.lua dissectors wiki.wireshark.org/Contrib.

To install them on Ubuntu:

mkdir -p $(HOME)/.local/lib/wireshark/plugins
cd $(HOME)/.local/lib/wireshark/plugins
wget "https://wiki.wireshark.org/Contrib?action=AttachFile&do=get&target=omci.lua"
wget "https://wiki.wireshark.org/Contrib?action=AttachFile&do=get&target=BinDecHex.lua"

IMPORTANT - Apply this fix to BinDecHex.lua.

To capture the OMCI packets for all ONUs:

cat openonu.logs | grep -E "TxOmciMessage|RxOmciMessage" | grep msg > packets.trace

To capture the OMCI packets for a particular ONU:

cat openonu.logs | grep -E "TxOmciMessage|RxOmciMessage" | grep msg | grep [deviceId] > packets.trace

Once you have the omci.dump file you need to prepare it to be imported in wireshark with this command:

awk -F"OmciMessage" '/OmciMessage/{print $2}' packets.trace | cut -f3 -d'"' > rawdump
cat rawdump | sed -e 's/.\{2\}/& /g' | sed -e 's/^/000000 /' > omci.hex

And then in wireshark:

  • select File -> Import from Hex Dump

  • select Encapsulation Type -> Ethernet

  • set Ethernet -> Ethertype (hex): 88b5