Documenting the Format of DJI Log Files

DJI has not publicly documented the format of their log files. However, we have reverse engineered the format of DJI's ".txt" log files, which are recorded by the controller (usually in a smartphone app) during each flight. Despite the filename suffix ".txt", these are not text files. Instead, they are binary files (that contain very little 'text').

Format of the ".txt" log file

All multi-byte numeric values are little-endian.

The file is laid out as follows:

For files generated by DJI's "GO" app:

A 100-byte header
(This was smaller in very old versions of the log file.)
A records area
A details area

For files generated by DJI's "Fly" app (for the Mavic Mini):

A 100-byte header
A details area
A records area

The parsing of a ".txt" file is illustrated by the source code for the "djiparsetxt" application, described below.

Format of the header

The first 8 bytes of the file are a byte count that point past the end of the records area.
(Or it might just be the first 4 bytes of the file. But the next 4 bytes always seem to be zero, and multi-byte words are stored in little-endian order, so the two descriptions are equivalent.)
The next 2 bytes of the file are the length of the details area.
The next byte appears to indicate the app and/or OS version. Some known values are:
1. GO iOS
2. GO Android
3. GO
4. GO Android
5. GO 4 iOS
6. GO 4 Android (ca. 4.0?)
7. GO 4 Android (ca. 4.2.0?)
8. GO 4 Android (circa 4.2.8?)
9. Fly
10. Fly (Android, version 1.2.4), also DJI Mini 3 RC; incompatible file format, different from that described here!
The next byte is unknown
The remaining 88 bytes of the header (not present in very old versions of the log file) are all zero.

Format of the records area

The "records area" consists of a sequence of records. Each record consists of:

A 1-byte record type
A 1-byte record length
"record length" bytes of record payload
A 1-byte 'end of record' marker, with the value 0xFF.
(This 'end of record' marker is redundant, because of the existing record length byte.)
(JPEG records - described later - are somewhat different; they don't have the 0xFF 'end of record' marker, and the record length is ignored.)

The parsing of a record is illustrated by the source code for the "parseRecord()" function.

Format of the record payload

Payload scrambling

In recent versions of the log format, the contents of each record payload' have been obfuscated - i.e., scrambled, using an XOR (i.e., exclusive-OR)-based mechanism. It's not obvious why DJI has chosen to do this. Perhaps they are concerned about people falsifying the contents of these logs, and chose to scramble the record payloads to try to make this harder? (However, a much better way to prevent people modifying the contents of logs would have been to add a digital signature, leaving the payloads in 'plaintext'.)

Fortunately, we are able to unscramble these payloads, as follows:

The first byte of each record payload is combined with the record type to generate an array of 8 pseudo-random bytes, which we call scramble bytes. (The 8 scramble bytes are generated by the function getScrambleBytes(), using a 64-bit CRC.)

The remaining (i.e., "record length-1") bytes of the payload are unscrambled by XORing them with the scramble bytes, as follows:

    for (int i = 1; i < record_length; ++i) {
        unscrambled_payload[i-1] = record_payload[i] ^ scramble_bytes[(i-1)%8];
    }

The resulting unscrambled_payload (length: record length-1) is interpreted, depending on the record type, as follows:

OSD
HOME
GIMBAL
RC
CUSTOM
DEFORM
CENTER_BATTERY
SMART_BATTERY
APP_TIP
APP_WARN
RC_GPS - format unknown
RC_DEBUG - format unknown
RECOVER
APP_GPS
FIRMWARE
OFDM_DEBUG - format unknown
VISION_GROUP - format unknown
VISION_WARN - format unknown
MC_PARAM - format unknown
APP_OPERATION - format unknown
(–23.) unknown
APP_SER_WARN
(–39.) unknown
COMPONENT
(–56.) unknown
JPEG - format described below
(–254.) unknown
not used

JPEG records are formatted differently from the others. They are not scrambled, and do not end with a 0xFF byte. Also, the record length byte, although present, is not used.

After two initial 'zero' bytes, a JPEG record consists of a sequence of (zero or more) JPEG images. Each JPEG image begins with a JPEG SOI ("start of image") marker: 0xFF 0xD8, and ends with a JPEG EOI ("end of image") marker: 0xFF 0xD9. An EOI marker that is not immediately followed by a SOI marker (beginning another JPEG image) marks the end of the JPEG record.

Our "djiparsetxt" application - described below - automatically extracts (into ".jpg" files) any JPEG images that it finds within JPEG records.

Format of the details area

The format of the details area is illustrated by the source code for the "parseDetails()" function.

Problems to be solved

If you can help resolve any of these, please let us know on our public mailing list, described below.

There are several record types that have been given a name, but which we do not currently know how their payloads are formatted. These include:
1. (==0x0B) "RC_GPS"
2. (==0x0C) "RC_DEBUG"
3. (==0x10) "OFDM_DEBUG"
4. (==0x11) "VISION_GROUP"
5. (==0x12) "VISION_WARN"
6. (==0x13) "MC_PARAM"
7. (==0x14) "APP_OPERATION"
We have seen other record types for which we do not know neither the name nor the format. These include:
1. (==0x16)
2. (==0x19)
3. (==0x1a)
4. (==0x1e)

The "djiparsetxt" application

"djiparsetxt" is a command-line application that reads a DJI '.txt' file and outputs - to stdout - the contents of the file in comma-separated value (i.e., "CSV") form. This output can then be used as input to a spreadsheet, or it could be piped into another application.

The application is provided as (C++) source code; not as a pre-built binary application. To use it, you must:

Have basic software development tools ("tar", "make", a C++ compiler)
Be familiar with running (Unix, Linux, or Mac OS) command-line applications, and understand I/O redirection (including the terms "stdout" and "stderr").
If you wish to modify the source code (e.g., to change which columns get included in the 'CSV' output, and/or change the order of columns), you must have at least a basic understanding of C++. (Note also that the source code is copyrighted and released under version 3 of the GNU GPL; you should understand your obligations under this license.)

The application's source code is available - as the file "djiparsetxt-latest.tar.gz" - here. Use "tar -xf" to extract the code; then cd to the source code directory. Then run: make

Running the "djiparsetxt" application

The simplest way to run "djiparsetxt" is to use it to convert a DJI '.txt' log file into a CSV (i.e., comma-separated value) file that you can view in a spreadsheet:

    djiparsetxt 'input-txt-file-name' > output-csv-file-name

E.g., if your input log file is named "test.txt", you could run:

    djiparsetxt 'test.txt' > test.csv

In this particular case, the single-quote marks (') around the input file name aren't necessary. However, DJI log file names often contain square-bracket characters ([ and ]) that will need to be 'quoted' - e.g.

    djiparsetxt 'DJIFlightRecord_2017-09-01_[16-42-09].txt' > test.csv

Note that in addition to outputting a CSV file to 'stdout', the program also outputs lots of diagnostic information to 'stderr'. You can usually just ignore this.

If you wish to change the fields that are output to the CSV file (e.g., change the number of output fields, and/or their order), then you could do so by modifying the code for the outputOneRow() function.

The "generateKML" application

"generateKML" is a shell script - bundled along with the "djiparsetxt" software - that reads a CSV file (from 'stdin'), and converts it to a KML-format file that can be read by "Google Earth" to display the drone's track.

"generateKML" is written in the Tcl programming (scripting) language. To run it, you must have Tcl (including the "tclsh" shell) installed on your system; you can get this software here, or using your OS's package/port system.

Running the "generateKML" application

The simplest way to run "generateKML" is to use it to convert a CSV file (that you previously created by running "djiparsetxt") into a KML-format file (that you can then view using Google Earth) - e.g.

    generateKML < test.csv > test.kml

Alternatively you can avoid creating a file "test.csv", by piping the output from "djiparsetxt" directly to the input of "generateKML" - e.g.,

    djiparsetxt 'test.txt' | generateKML > test.kml

The "dji-log-discuss@lists.live555.com" mailing list

We have a public mailing list

  dji-log-discuss@lists.live555.com

for discussion of DJI's log file formats (and the "djiparsetxt" application). Note, however, that before you can post to the mailing list, you must first subscribe to it. (This is standard for all Internet mailing lists; it helps protect against spam.)

(Note that you must subscribe to the mailing list using the same "From:" address that intend to use to later post messages to the list.)

Thanks

Many thanks to the authors of the following document and software, which were helpful for this reverse engineering:

"DROP (DRone Open source Parser) your drone: Forensic analysis of the DJI Phantom III" (by Clark, Meffert, Baggili, and Breitinger)
"dji-log-parser", by 'mikeemoo' (a parser for an earlier version of DJI's '.txt' log file)