Input Data files¶

You need a frequency file which contains the frequencies, each in a seperate line, in ascending order:

frequency.dat (Download frequencies.dat (unix)) (Download frequencies.dat (Windows)):

Complex resistivity spectra are provided using a data file which holds a spectrum in each line. Columns are separated by space and values are linear both for magnitude and phase values. The first N columns correspond to the magnitude values ( $\Omega m$ ) corresponding to the frequencies stored in frequencies.dat. The following N columns represent the corresponding phase values.

data.dat (download data.dat (unix))

(download data.dat (Windows))

49.345594 49.120225 48.860658 48.589371 48.333505 48.113950 47.939222\
47.807051 47.709583 47.637735 47.583349 47.539704 47.501267 47.463162\
47.420588 47.368190 47.299354 47.205358 47.074354 46.890271 46.632118\
46.274900 45.794402 45.178163 44.441082 -10.526822 -12.095446 -13.004975\
-12.999086 -12.088092 -10.544173 -8.744458 -7.007706 -5.526119 -4.380307\
-3.584099 -3.124956 -2.990678 -3.184856 -3.735642 -4.701107 -6.172278\
-8.272438 -11.148023 -14.941904 -19.734922 -25.441545 -31.665354\
-37.581057 -41.99903

Note

The previous listing for the data.dat file contains only one line. For display purposes, line breaks were introduced, and indicated by ‘\’ characters.

Input data formats¶

Input data in data.dat can be specified in various formats (internally converted using the sip_formats.convert) module:

lnrmag_rpha: The first N columns are resistivity magnitudes in the natural logarithm; the second N columns contain resistivity phase values (mrad)
log10rmag_rpha: The first N columns are resistivity magnitudes in the base-10 logarithm; the second N columns contain resistivity phase values (mrad)
rmag_rpha: The first N columns are linear resistivity magnitudes; the second N columns contain resistivity phase values (mrad)
rre_rim: The first N columns contain resistivity real parts; the second N columns contain resistivity imaginary parts
rre_rmim: The first N columns contain resistivity real parts; the second N columns contain the negative of resistivity imaginary parts
cmag_cpha: The first N columns contain conductivity magnitudes, the second N columns contain conductivity phase values
cre_cim: The first N columns contain conductivity real parts; the second N columns contain conductivity imaginary parts
cre_cmim: The first N columns contain conductivity real parts; the second N columns contain the negative of conductivity imaginary parts

N is the number of frequencies. Specify the data format using the --data_format command line option. For example, to use conductivity real and imaginary parts as input data, use:

ccd_tools --data_format cre_cim ...

Output Files¶

The following sections describe the output files of the various formats. Formats can be changed using the --output_format switch.

ascii_audit format¶

This output format saves fit results to ASCII text files. Each text file contains a header which uniquely identifies the fit run that the results were created from. In addition, this format tries to group results into one file, where possible. Using the header information, clear process logs can be created, and mix-ups of results can be prevented.

This is the default output format.

Note

Can be enabled using the --output_format ascii_audit switch

Note

While this output format is great for ensuring a clear processing trail, it is currently not possible to directly use the OUTPUT files as INPUT files for new decompositions!

header¶

Each output file consists of two parts: the header (four lines), and the data part (the rest of the file). The header looks like this

# id:30ffcd8e-2b75-48f9-acc4-3f93c9c0d48e
# 20150730_14h:32m
# /home/mweigand/.local/bin/cdd_time -f data/frequencies.dat --times data/times.dat\
    -d data/data.dat -o results_time --f_lambda 50\
    --tmi_first_order --tm_i_lambda 5000 --plot
# covf

the ‘#’ character indicates a non-data line
the first line contains a unique id, which is generated for each run of either ccd_single or ccd_time. This id changes also for repeated runs of the same settings!
The second line contains a datetime string: first the date in the order year-month-day, followed by the hour (24 hours) and minute.
The third line contains the command call. Note that in the example above, the command is stretched over three lines. This is only for display purposes, and in the files will always be in one line.
The fourth line contains the data header, indicating the type of data (sometimes even column descriptions). results/

Files¶

A typical result directory in this format contains the following files

|-- covf.dat
|-- covm.dat
|-- data.dat
|-- errors.dat
|-- f.dat
|-- frequencies.dat
|-- integrated_paramaters.dat
|-- inversion_options.json
|-- lams_and_nr_its.dat
|-- m_i.dat
|-- normalization_factors.dat
|-- tau.dat
|-- times.dat
`-- version.dat

0 directories, 14 files

covf.dat: Frequency coverage values for all spectra. Each line contains the data for one spectrum, with N columns corresponding to the N frequency values of covf.

covm.dat: Parameter coverage values for all spectra. Each line contains the data for one spectrum, with M columns corresponding to the M model parameters (i.e. $\rho_0$ and the $m_i$ values).

data.dat contains the input data saved in the format used by the chosen forward model. Each line contains one full spectrum with 2 * N columns, with N being the number of frequencies. If the resistivity formulation is used for the decomposition process, the first N columns contain the real parts of the resistivities, if the conductivity formulation is used, real parts of the conductivities are stored here. Correspondingly, the second N columns contain imaginary parts of either the resistivities, or the conductivities.

errors.dat: Stores the data weighting errors from the matrix $\underline{\underline{W}}_d$ .

The forward response of the final iteration is stored in f.dat. Each row contains one spectrum, with the first N columns containing either magnitudes or real parts, corresponding to the N frequencies. The second N columns contain either phase values or imaginary parts. Which parameters are stored in the column blocks is defined in the output file f_format.dat.
frequencies.dat: Frequencies [Hz], each line contains one frequency, in ascending order.
integrated_paramaters.dat: Contains the integrated parameters as computed from the RTD. Each line contains the parameters of one spectrum, with the columns decribed in the header.
inversion_options.json*: inversion options (as set by command line options, primarily for debug purposes)
lams_and_nr_its.dat:
m_i.dat: RTDs for all spectra. Each line contains the RTD of one spectrum.
normalization_factors.dat: Normalization factors used (will only be created if the --norm option was activated.

tau.dat: stores the relaxation times used for decomposition
times.dat: times corresponding to the spectra (only created by ccd_time

version.dat: This file contains the package versions of either dd_tools, or geccoinv and dd_interface (depending on the installation method, dd_tools groups the later two packages).

ascii format¶

This output format saves the files to ASCII text files. Most parameters/variables are saved to a separate file, resulting in a large number of files. No, or only very limited, headers are provided. This format can make postprocessing easy, but should only be used if really necessary.

Note

can be enabled using the --output_format ascii switch

A typical result directory in this format contains the following files

results/
|-- command.dat
|-- data.dat
|-- data_format.dat
|-- errors.dat
|-- f.dat
|-- f_format.dat
|-- frequencies.dat
|-- inversion_options.json
|-- lambdas.dat
|-- normalization_factors.dat
|-- nr_iterations.dat
|-- omega.dat
|-- rms_definition.json
|-- s.dat
|-- stats_and_rms
|   |-- covf_results.dat
|   |-- covm_results.dat
|   |-- decade_bins_results.dat
|   |-- decade_loadings_results.dat
|   |-- f_50_results.dat
|   |-- f_arithmetic_results.dat
|   |-- f_geometric_results.dat
|   |-- f_max_results.dat
|   |-- f_mean_results.dat
|   |-- f_peak1_results.dat
|   |-- f_peak2_results.dat
|   |-- f_peaks_all_results.dat
|   |-- m_data_results.dat
|   |-- m_i_results.dat
|   |-- m_tot_n_results.dat
|   |-- m_tot_results.dat
|   |-- rho0_results.dat
|   |-- rms_all_error.dat
|   |-- rms_all_noerr.dat
|   |-- rms_imag_parts_error.dat
|   |-- rms_imag_parts_noerr.dat
|   |-- rms_real_parts_error.dat
|   |-- rms_real_parts_noerr.dat
|   |-- tau_50_results.dat
|   |-- tau_arithmetic_results.dat
|   |-- tau_geometric_results.dat
|   |-- tau_max_results.dat
|   |-- tau_mean_results.dat
|   |-- tau_peak1_results.dat
|   |-- tau_peak2_results.dat
|   |-- tau_peaks_all_results.dat
|   `-- U_tau_results.dat
|-- tau.dat
`-- version.dat

1 directory, 48 files

The following output files will be created in the selected output directory. These files are described below, sorted by category.

Input data¶

data.dat contains the input data saved in the format used by the chosen forward model. Each line contains one full spectrum with 2 * N columns, with N being the number of frequencies. If the resistivity formulation is used for the decomposition process, the first N columns contain the real parts of the resistivities, if the conductivity formulation is used, real parts of the conductivities are stored here. Correspondingly, the second N columns contain imaginary parts of either the resistivities, or the conductivities.
data_format.dat contains the data format in the format usable with the --data_format command line option (usually cre_cim).
Frequencies and corresponding angular frequencies (
$\omega = 2 \cdot \pi \cdot f$ ) are stored in the files frequencies.dat and omega.dat.
The file command.dat holds the complete call to the fit routine
A JSON formatted file inversion_options.json stores internal inversion options. This file is mainly for debugging purposes, and needed to recreated inversion objects from fit results.
rho_normalizations.dat contains normalization factors if the option --norm_mag was used.

ccd_time only:

the file times.dat contains the time strings as read from the input files. One time per line.

Filter results¶

Note

Filtering using ddps.py is still an experimental feature and might not work at the moment.

filter_mask.dat contains the remaining indices after a filter operation with ddps.py

Primary fit results¶

$\tau$ and $s = log_{10}(\tau)$ values are stored in the files tau.dat and s.dat, respectively.
The regularization parameters of the last iterations are stored in the file lambdas.dat, one per line:
```
1.000000000000000021e-03
1.000000000000000056e-01
```
The chargeability values of the last iteration can be found in stats_and_rms/m_i_results.dat

The forward response of the final iteration is stored in f.dat. Each row contains one spectrum, with the first N columns containing either magnitudes or real parts, corresponding to the N frequencies. The second N columns contain either phase values or imaginary parts. Which parameters are stored in the column blocks is defined in the output file f_format.dat.

RMS values are stored in the subdirectory stats_and_rms, using the following files (final RMS of each spectrum per line). real/imag here correspond to real part and imaginary part of resistivity, respectively. The _error suffix denotes RMS values computed with data weighting.

filename	description
rms_all_error.dat	RMS of real and imaginary parts, including error weighting
rms_all_noerr.dat	RMS of real and imaginary parts, without error weighting
rms_imag_parts_error.dat	Error weighted RMS of imaginary parts
rms_imag_parts_noerr.dat	Non-error weighted RMS of imaginary parts
rms_real_parts_error.dat	Error weighted RMS of real parts
rms_real_parts_noerr.dat	Non-error weighted RMS of real parts

$RMS_{\text{no error}} = \sqrt{\frac{1}{N} \sum_i^N d_i - f_i(m)}\\ RMS_{\text{with error}} = \sqrt{\frac{1}{N} \sum_i^N \frac{d_i - f_i(m)}{\epsilon_i}}$

The number of iterations actually used for each spectrum are stored in nr_iterations.dat.

errors.dat: Stores the data weighting errors from the matrix $\underline{\underline{W}}_d$ .
Normalization factors are stored in normalization_factors.dat

version.dat: This file contains the package versions of either dd_tools, or geccoinv and dd_interface (depending on the installation method, dd_tools groups the later two packages).

Integral parameters¶

Statistical parameters are stored in the subdirectory stats_and_rms, and all output files have the same file format. Each line contains the value of one spectrum. This applies to the following files (N = number of frequencies, M = number of parameters):

filename	stored values per line
m_i_results.dat	$m(\tau_i)$
m_tot_n_results.dat	$log_{10}(m_{tot}^n)$
m_tot_results.dat	$log_{10}(m_{tot}^n)$
rho0_results.dat	$log_{10}(\rho_0)$
tau_50_results.dat	$log_{10}(\tau_{50})$
tau_mean_results.dat	$log_{10}(\tau_{mean})$
tau_arithmetic_results.dat	$log_{10}(\tau_{arithmetic})$
tau_geometric_results.dat	$log_{10}(\tau_{geometric})$
tau_peak1_results.dat	$log_{10}(\tau_{peak}^1)$
tau_peak2_results.dat	$log_{10}(\tau_{peak}^2)$
tau_peaks_all_results.dat	$log_{10}(\tau_{peak}^i)$
tau_x_.dat*	$log_{10}(\tau_x)$ ; see description below
tau_max.dat	$\tau$ corresponding to max. chargeability. First occurence.
U_tau_results.dat	Uniformity parameter $U_{\tau} = \frac{\tau_{60}}{\tau_{10}}$
f_50_results.dat	frequency corresponding to $\tau_{50}$
f_mean_results.dat	frequency corresponding to $\tau_{mean}$
f_peak1_results.dat	frequency corresponding to $\tau_{peak}^1$
f_peak2_results.dat	frequency corresponding to $\tau_{peak}^2$
f_peaks_all_results.dat	frequency corresponding to all peaks $\tau_{peak}^N$
covf_results.dat	coverage frequencies (N columns)
covm_results.dat	coverage parameters (M columns)

Integral parameters are explained in the section Integral Parameters.

Input Data files¶

Input data formats¶

Output Files¶

ascii_audit format¶

header¶

Files¶

ascii format¶

Input data¶

Filter results¶

Primary fit results¶

Integral parameters¶

Time-lapse Cole-Cole decomposition routines

Navigation

Related Topics