{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Results and Models\n", "\n", "This page provides an overview of Result objects with a view to key interactions and access to model outputs. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2\n", "%matplotlib inline\n", "import atomica as at\n", "import numpy as np\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First we will run a simulation to produce a `Result` object." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "P = at.demo('tb')\n", "result = P.results[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The plotting documentation discusses how to generate plots in detail. This documentation will thus focus on methods and functionality of the `Result` object itself, with a view to \n", "\n", "1. Generating pre-specified plots\n", "2. Accessing and interacting with raw values\n", "\n", "## Pre-specified plots\n", "\n", "It is possible to pre-define plots in the Framework by creating a sheet called 'Plots'. This sheet contains a specification of a set of plots to generate:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "P.framework.sheets['plots'][0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The most important columns here are the `name` which identifies the plot, and the `quantities` which corresponds to a valid input to the `outputs` argument of `PlotData`. To generate a plot, simply pass the name of the plot to `Result.plot`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.plot('New active TB infections');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The output is the same as if a `PlotData` object was constructed, a series plot generated with `axis='pops'`, and the title set to match the specified title:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "d = at.PlotData(result,outputs={'New incident cases':['leu_act:flow','llu_act:flow', 'lex_act:flow', 'llx_act:flow', ]})\n", "at.plot_series(d,axis='pops');\n", "plt.title('New active TB infections');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Calling `Result.plot()` with no arguments will result in all plots in the Framework being produced. At the moment, it is only possible to define series plots in the framework." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exporting results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Many key outputs are generated by aggregation of raw model quantities - for example, in the demo TB model, the number of new cases is given by aggregating together several different modes of latent activation. Thus, often the main quantities for analysis are those defined as plottable quantities on the Framework's 'Plots' sheet. \n", "\n", "These outputs can be written to an Excel file using the `at.export_results()` function. This is not a method of the `Result` because it is possible to export multiple results at the same time, which is shown later in this document. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "at.export_results(result,'example1.xlsx');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This produces a spreadsheet like the one shown below:\n", "\n", "![resultsexport1](assets/results_export1.jpg)\n", "\n", "Additional sheets will be present showing with numerical values for any cascades that are present:\n", "\n", "![resultsexport1cascades](assets/results_export1_cascades.jpg)\n", "\n", "If any parameters are targetable by programs, the exported results will also include the value of those parameters, which facilitates monitoring program overwrites. \n", "\n", "![resultsexport1targets](assets/results_export1_targets.jpg)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If programs are present, an additional sheet will be written with program-related outputs. To demonstrate this, we will now first generate some scenario results:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "at.make_demo_scenarios(P)\n", "scen_results = P.run_scenarios()\n", "at.export_results(scen_results[0],'example2.xlsx');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have now exported results for a single simulation with programs active. The exported spreadsheet now contains a 'Programs' sheet with contents like:\n", "\n", "![resultsexport2programs](assets/results_export2_programs.jpg)\n", "\n", "We next turn to the question of exporting multiple results. The scenario set that we ran above produced 3 results. We can write all three to a workbook by passing all of them to `at.export_results`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "at.export_results(scen_results,'example3.xlsx');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This produces a workbook where all of the results are displayed together. For example, the original 'Plot data' sheet now shows:\n", "\n", "![resultsexport3plotdata](assets/results_export3_plotdata.jpg)\n", "\n", "with similar changes to the other sheets as well.\n", "\n", "By default, the results are shown with each quantity (e.g. parameter, program) being in a separate table, that is grouped by result, then by population. So in the screenshot above, the table is for the 'Population size' quantity, on the left there are blocks for the 'Default budget' scenario, the 'Doubled budget' scenario etc. and then within each block, the population are shown. \n", "\n", "Sometimes it is useful to group the quantities differently. For example, if we wanted to compare the number of people in each of the populations under each scenario. To do this, simply specify a different grouping order. The default grouping order described and shown above is `('output','result','pop')`. If we wanted to place all of the same populations together, specify the order as `('output','pop','result')` instead." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "at.export_results(scen_results,'example4.xlsx',output_ordering=('output','pop','result'));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This produces the following output\n", "\n", "![resultsexport4plotdata](assets/results_export4_plotdata.jpg)\n", "\n", "Notice how the values are grouped differently now. The ordering is specified independently for the plot data, cascades, and programs - see the documentation for the `export_results` function for full details. \n", "\n", "Finally, note that it was only necessary to pass a list of results to `at.export_results` - this means that it is possible to combine arbitrary results into a single spreadsheet as required (they do not all have to come from the same set of scenarios or optimization). " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exporting raw results\n", "\n", "The model contains a raw representation of the values at each timestep associated with parameters, compartments, characteristics, and flow rates (links). These raw values can be exported using the `Result.export_raw()` method of the result object. This method dumps the raw output for a single result. For example:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.export_raw('export5.xlsx');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This produces an Excel file with the following content:\n", "\n", "![resultsexport5a](assets/results_export5a.jpg)\n", "\n", "Some rows have been hidden for clarity (see row numbers above). The raw output is quite detailed - for the TB example model, there are over 2000 rows of output.\n", "\n", "In addition, `Result.export_raw()` returns a Pandas `DataFrame` with the contents that get written to the file - in fact, if you provide a filename when calling `export_raw()`, the only difference is that `export_raw()` automatically writes the returned dataframe to the specified file. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = result.export_raw();\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using the `DataFrame` can provide a convenient way to perform extra operations or to customize the spreadsheet. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Accessing raw values\n", "\n", "Within the `Result` object, a `Model` object stores all of the underlying quantities. Within the `Model`, they are stored hierarchically within `Population` objects. For example:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.model.pops" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.model.pops[0].comps[0:5]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.model.pops[0].pars[0:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each of these objects has a `vals` attribute that stores the actual values. For convenience, they also have a `t` attribute that stores the time values" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.model.pops[0].comps[1].t" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.model.pops[0].comps[1].vals" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These objects have a `.plot` method that can be used to generate a diagnostic plot - this is mainly useful for debugging" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.model.pops[0].comps[1].plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can retrieve a specific population and object by name rather than by index" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.model.get_pop('0-4').get_comp('sus')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Methods `get_comp`, `get_par`, `get_charac` and `get_links` exist to retrieve variables of a specific type. If you have a code name but don't know the type of variable, you can use the generic `get_variable` method. This method returns a list of matching objects (because if the code name corresponds to a link, there could be more than one matching object)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.model.get_pop('0-4').get_variable('sus')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `get_variable` method also supports the standard link syntax `source:destination:par_name` e.g." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.model.get_pop('0-4').get_variable(':ddis')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For convenience, the `Result` object has a `get_variable` method that wraps accessing `get_pop` and `get_variable` for the underlying `Model` object. So for example, the most convenient way of looking up a particular quantity is with:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.get_variable('sus','0-4')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The population name is optional in `Result.get_variable` - if you omit it, it will return a list of all matching variables in all populations. This can be useful if there are multiple population types, and you don't know which populations contain a particular variable." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result.get_variable('sus')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Traversing the graph\n", "\n", "The integration objects (such as `Compartment`, `Parameter` etc.) are very powerful because they store information about the relationship between quantities in the computational graph, which is not present if simply working with the values. These relationships are discussed in detail in the documentation of the model internals, but the most useful quantities are\n", "\n", "- `Compartment.inlinks` - All links flowing into the compartment\n", "- `Compartment.outlink` - All links flowing out of a compartment\n", "- `Link.source` - The source compartment for the link\n", "- `Link.dest` - The destination compartment for the link\n", "- `Link.parameter` - The parameter supplying values for the transition\n", "- `Parameter.links` - All of the links associated with the the parameter\n", "- `Parameter.deps` - A dict with all of the parameter's dependencies required to evaluate its function\n", "- `Characteristic.includes` - All of the included compartments in a characteristic\n", "- `Characteristic.denominator` - The denominator compartment or characteristic\n", "\n", "For example" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "comp = result.get_variable('sus','0-4')[0]\n", "comp" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "comp.outlinks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Crucially, _all of these variables are references to other objects_. Therefore, it's possible to chain the lookups together. For example, if we wanted to access the parameter governing the flow from `sus` to `vac` out of the `sus` compartment, we could use" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "link = comp.outlinks[0] # Get a link flowing out of the `sus` compartment\n", "par = link.parameter # Get the parameter associated with that link\n", "par" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This functionality can be extremely useful in exploratory analysis because it makes it easy to trace through different parts of the model." ] } ], "metadata": { "kernelspec": { "display_name": "Python [conda env:atomica37]", "language": "python", "name": "conda-env-atomica37-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" }, "pycharm": { "stem_cell": { "cell_type": "raw", "metadata": { "collapsed": false }, "source": [] } }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }