Differences

This shows you the differences between two versions of the page.

--- calibrating_the_global_trade_model [2020/03/01 07:45] – created matsz
+++ calibrating_the_global_trade_model [2022/11/07 10:23] (current) – external edit 127.0.0.1
@@ Line 25: / Line 25: @@
 Constraints, requirements, policies and other data loaded including base year and trends data (i.e., of res_BBCC.gdx  and trends_BBYY.gdx files) are subject to certain (mainly non-major) adjustments, additional calculations and assumptions that serve the purposes of data balancing, checking and provision of necessary for the calibration information. These include, for example, deleting positions not needed during the calibration run, (re-)assigning parameter names, deleting tiny quantities, checks for production without activity levels, possible empty projections and negative inland waters, setting the output coefficients for young animals equal to the ones at country (EU MSs, as young animals are not presented in the non-EU countries) level if missing at regional level, correcting fat and protein content of raw milk, assumption that second generation biofuels are produced 50/50 by agricultural residuals and new energy crops, etc.
-Next, FAO data on the non-European countries as well as the trade flows among all of the countries (country trade blocks) accounted for in CAPRI are loaded. These FAO data together with the European data, which has already been subjected to certain adjustments as described in the previous paragraph, undergo the, so-called, data preparation step. This process is controlled by C:\...\CAPRI\gams\arm\market1.gms file which calls the C:\...\CAPRI\gams\arm\data_prep.gms file - specifically for this step. The data preparation step mostly refers to the base year and includes: among else, modification of GDP to fit the sum of final household expenditure, final government expenditure, gross capital formation and current account balance; import and export flows to be in line with net trade from production minus demand; scaling of demand side to fit production plus net trade; estimation of consumer prices for some countries, if missing; calculation of nutrient consumption per head and day as net of losses in distribution and households; scaling of outliers in prices etc. This step as well provides with estimation of yearly change factors beyond the base year: for prices, GDP, population, quantities and areas. Additionally, i) substitution elasticities (i.e., p_rhoX, where X indicates continuation of the parameter name) for bio-fuel feedstocks, feed, dairy products, sugar, table grapes, tobacco, cheese, fresh milk products, fruits, vegetables, distilled dried grains and rice for the CAPRI demand system((See FIXME section 5.4.1 on Overview of the market model "CAPRI comprises a two stage Armington system: on the top level, the composition of total demand from imports and domestic sales is determined, as a function of the relation between the domestic price and the average import price. The lower stage determines the import shares from different origins and defines the average import price.")), and ii) transformation elasticity for oil seed processing and land supply elasticities are assigned.
+Next, FAO data on the non-European countries as well as the trade flows among all of the countries (country trade blocks) accounted for in CAPRI are loaded. These FAO data together with the European data, which has already been subjected to certain adjustments as described in the previous paragraph, undergo the, so-called, data preparation step. This process is controlled by C:\...\CAPRI\gams\arm\market1.gms file which calls the C:\...\CAPRI\gams\arm\data_prep.gms file - specifically for this step. The data preparation step mostly refers to the base year and includes: among else, modification of GDP to fit the sum of final household expenditure, final government expenditure, gross capital formation and current account balance; import and export flows to be in line with net trade from production minus demand; scaling of demand side to fit production plus net trade; estimation of consumer prices for some countries, if missing; calculation of nutrient consumption per head and day as net of losses in distribution and households; scaling of outliers in prices etc. This step as well provides with estimation of yearly change factors beyond the base year: for prices, GDP, population, quantities and areas. Additionally, i) substitution elasticities (i.e., p_rhoX, where X indicates continuation of the parameter name) for bio-fuel feedstocks, feed, dairy products, sugar, table grapes, tobacco, cheese, fresh milk products, fruits, vegetables, distilled dried grains and rice for the CAPRI demand system((See section [[Market module for agricultural outputs#Overview on the market model]] on Overview of the market model "CAPRI comprises a two stage Armington system: on the top level, the composition of total demand from imports and domestic sales is determined, as a function of the relation between the domestic price and the average import price. The lower stage determines the import shares from different origins and defines the average import price.")), and ii) transformation elasticity for oil seed processing and land supply elasticities are assigned.
-Together with the data, equations of the CAPRI market module are loaded. They are described in detail in FIXME section 5.4. These equations include behavioural functions for market demand including expenditure function, feed demand, blocks for dairy products, oilseeds processing and biofuels, netput functions, trade equations and balances, equations for prices and price transmission, functions for trade policies and for intervention stocks. There are additionally two crucial for data calibration functions: minimization of deviation of estimated values from the observed data. These two functions are described in detail later in this section.
+Together with the data, equations of the CAPRI market module are loaded. They are described in detail in section [[Market module for agricultural outputs]]. These equations include behavioural functions for market demand including expenditure function, feed demand, blocks for dairy products, oilseeds processing and biofuels, netput functions, trade equations and balances, equations for prices and price transmission, functions for trade policies and for intervention stocks. There are additionally two crucial for data calibration functions: minimization of deviation of estimated values from the observed data. These two functions are described in detail later in this section.
 ===Data balancing===
@@ Line 37: / Line 37: @@
 Data calibration for the base year aims at modifying the base year data to fit the system of equations of the market module. Some of the parameters defined in Stage I (e.g., p_rhoX) as well as parameter values and bounds defined at this stage are used. For example, starting points and corridors for quantity variables are set (e.g., calculating of world production to define correction corridor for calibration of production/demand/trade flows globally), global TRQ data are converted into ad valorem tariffs and checked for consistency and completeness, policy variables for the EU market model such as e.g., intervention stocks, are loaded. Also, starting values for prices of dairy products are estimated. In particular, a non-linear programming model is used, where the objective function is formulated as a Highest Posterior Density function. The value of this objective function equals sum of squared deviations of fat and protein prices, fat and protein content of milk products and processing margins of milk products from the respective means, weighted with the a priori variances. The means are defined as parameters based on the prices and fat and protein content of milk in the base year. The objective function is restricted by the balance: fat and protein of raw milk delivered to dairies shall equal fat and protein content of dairy products. The model is solved by minimizing the value of highest posterior density, hence minimizing the differences between the variables and their means. Prices of milk products are then defined as: product of fat and protein content and of far and protein prices plus processing margin. Furthermore, administrative prices for cereals and dairy products, and minimal import prices for cereals are constructed.
-With the file C:\...\CAPRI\gams\arm\cal_models.gms, the so-called, models, used in calibration of data base are defined and solved. These models represent collection of equations, solutions of which provide with parameter values used for data calibration. The first model (MODEL m_trimSubsExports) calibrates the parameters of the function which defines the values of subsidized exports with and without the increase of market price above the administrative price. The second model (MODEL m_trimInterv) defines parameters of equations for intervention stock changes. It includes an objective function defined as a sum of: squared scaled difference of estimated and observed intervention stock changes and squared scaled parameters for behavioural function of intervention stock changes. This objective function is minimized subject to constraints represented by equations for intervention sales, probability for an undercut of administrative price, release from intervention stock, intervention stock changes and value of the intervention stock. The constraints are equations of the market model ( FIXME see section 5.4).
+With the file C:\...\CAPRI\gams\arm\cal_models.gms, the so-called, models, used in calibration of data base are defined and solved. These models represent collection of equations, solutions of which provide with parameter values used for data calibration. The first model (MODEL m_trimSubsExports) calibrates the parameters of the function which defines the values of subsidized exports with and without the increase of market price above the administrative price. The second model (MODEL m_trimInterv) defines parameters of equations for intervention stock changes. It includes an objective function defined as a sum of: squared scaled difference of estimated and observed intervention stock changes and squared scaled parameters for behavioural function of intervention stock changes. This objective function is minimized subject to constraints represented by equations for intervention sales, probability for an undercut of administrative price, release from intervention stock, intervention stock changes and value of the intervention stock. The constraints are equations of the market model (see section [[Market module for agricultural outputs]]).
 The model that calibrates base year data (MODEL m_calMarketBas) is defined in cal_models.gms file as well and includes almost all equations of the market model. In particular: equations for processing margin for dairy products (ProcMargM_), fat and protein balance between raw milk and dairy products (FatsProtBal_), processing margin for oilseeds ProcMargO_, processing yields of oilseeds (procYield_), 1st generation output of biofuels (prodBiof_) and total output of biofuels (MaprBiof_); __balancing and adding up equations__: equations which add production, processing demand, human consumption, feed demand quantities and quantities for processing from single countries (or block of countries) to trade blocks (ProdA_, ProcA_, HconA_, FeedUseA_, Proca_), adding up inside of the Armginton aggregate (total domestic consumption) (ArmBal1_), supply balance (SupBalM_) and imports and exports added up to bilateral trade flows (excluding diagonal element) (impQuant_); __price equations__: 1st stage Armington quantity aggregate (ArmFit1_), 2nd stage Armington quantity aggregate (ArmFit2_), import price relation to producer price (impPrice_), consumer price as average of domestic and import prices (arm1Price_), average price as average of different import prices (arm2Price_), average import price (arm2Val_), consumer price (Cpri_), producer price (PPri_), market price (PMrk_), average market price (MarketPriceAgg_); __trade and tariff equations__: aggregated trade flows (TradeFlowsAgg_), average transportation costs (TransportCostsAgg_), sum of imports under a non-allocated TRQ (TRQImports_), share of the tariff applied for the EU entry price system (EntryPriceDriver_), tariff specific entry price (tarSpecIfEntryPrice_), Cif price (cifPrice_), equation for defining levy (replaces tariff) in case of minimal border prices (FlexLevyNotCut_), cuting flexible levy by specific tariff if it exceeds the bound rate (FlexLevy_), tariffs under bi-lateral TRQs (trqSigmoidFunc_), specific tariffs as function of import quantities, if TRQ is present (tarSpec_, prefTriggerPrice_), tariffs under globally open (not bilaterally allocated) TRQs (tarSpecW_), ad valorem tariffs, if TRQ is present (tarAdval_), ad valorem tariff under not bilaterally allocated TRQs (tarAdValW_), export quantities from bi-lateral trade flows (expQuant_), exports included in the calculation of the export unit values excluding flows under double-zero agreements (nonDoubleZeroExports_), unit value exports (unitValueExports_, valSubsExports_), subsidised export values (EXPs_); __equations for intervention stocks__: probability weight for an undercut of administrative price (probMarketPriceUnderSafetyNet_), intervention sales (buyingToIntervStock_), intervention stock end size (intervStockLevel_), intervention stock changes (intervStockChange_), release from intervention stocks (releaseFromIntervStock_), aggregators for intervention purchases; equation for world market price (wldPrice_), and equation for minimization of deviation from given base year data and estimated data (NSSQ_). The model is solved by minimizing the SSQ value of NSSQ equation which is constrained by all of the rest of the equations included in the model.
@@ Line 50: / Line 50: @@
 where SSQ is an artificial variable to be minimized, indices RMS, XXX, BAS and i indicate, respectively, regions, commodities, base year and activities (e.g., production, processing, imports etc.), and p_weight is a parameter of weights between 1 and 100 assigned to regions and activities. These weights are necessary to achieve plausible calibrated values and their specification is the outcome of a trial and error process, inspecting results from data calibration and retrying. They depend on the results of global database and trends generation. On the right hand-side of the equation v stands for a variable to be estimated and DATA – for base year data already adjusted at the data preparation and balancing stage. Hence with this equation squared sum over regions and commodities of differences between estimated and observed values (and or quantities), these differences being scaled by the observed data times the weight parameters, is minimized. Respectively, calibrated base year data fits the system of the market equations, given certain parameter values, and resembles the observed data as closely as possible. The activities implied under the I index include quantities of production, human consumption, feed, processing, processed to biofuels, import and export, producer, consumer and market prices, difference between market prices and import prices to reduce differences between physical and Armington aggregation, consolidated gap between producer and market prices, processing margin, trade flows and transport costs.
-The process of model solving is navigated with C:\...\CAPRI\gams\arm\data_fit.gms file. Its main function is to assure model solving by keeping the market balances closed and price system consistent. Because of the very large number of equations with the exact similar number of variables (36 thsds) that makes the system of equations square, as well as non-linear formulation of some of the equations, it is very likely that infeasibilities will occur during the model solving. To ensure the feasibility as far as possible, code elements such as widening of variable bounds, once they become binding, reducing non-smoothness of the functional forms and introduction of slack variables are introduced. More detailed information on this process can be found in a technical document by Wolfgang Britz and Heinz-Peter Witzke I//nfeasibilities in the market model of CAPRI – how they are dealt with// at [[https://www.capri-model.org/docs/infes.pdf]].
+The process of model solving is navigated with C:\...\CAPRI\gams\arm\data_fit.gms file. Its main function is to assure model solving by keeping the market balances closed and price system consistent. Because of the very large number of equations with the exact similar number of variables (36 thsds) that makes the system of equations square, as well as non-linear formulation of some of the equations, it is very likely that infeasibilities will occur during the model solving. To ensure the feasibility as far as possible, code elements such as widening of variable bounds, once they become binding, reducing non-smoothness of the functional forms and introduction of slack variables are introduced. More detailed information on this process can be found in a technical document by Wolfgang Britz and Heinz-Peter Witzke //Infeasibilities in the market model of CAPRI – how they are dealt with// at [[https://www.capri-model.org/docs/infes.pdf]].
 After solving the MODEL m_calMarketBas, the calibrated data are stored, new producer prices for agricultural outputs are set, sugar beet prices as a function of – sugar market price – sugar export price (pre-reform) or ethanol market price (post-reform) – processing yield (specific to CUR to calibrate to any set of projected beet prices) – levying model for A- and B- sugar (pre-reform) are calculated, share and shift parameters of CES-functions used in the Armington approach to determine import shares as a function of import prices are defined (file C:\...\CAPRI\gams\arm\cal_armington.gms). Furthermore, energy conversion factors for animal products are defined with MODEL m_fitFeedConv (in file C:\...\CAPRI\ gams\arm\feed_conv_decl.gms).
@@ Line 87: / Line 87: @@
 At the final stage, some of the starting values and bounds for the market model are set, and agricultural policy data are loaded, adjusted and extended to the simulation year. The policy data include single area payment scheme, set-aside regulations, differentiation between old and new MSs payments, special national envelopes, Nordic schemes, changes in administrative prices, rural development policy and other major CAP post-2014 instruments. Policy files used for the baseline are located in C:\...\CAPRI\ gams\scen\base_scenarios folder. Their loading into the baseline process is controlled by CAP_2014_2020.gms file. With the data mentioned, the outcome of calibration of the CAPRI market module can be tested. In particular, the market model is solved at "trend values" and, thus, the calibration outcome is checked for fitting to the square system of market model equations. This is controlled by C:\...\CAPRI\.gams\arm\prep_market.gms file.
+====Technical remarks====
+Note that the task "Baseline calibration of market model" deletes the sim_ini.gdx file, but does not create a new one at the end of the calibration process. The new sim_ini.gdx file will be only created at the first simulation run after the calibration. That is also the reason why a specific GUI option 'Kill simini file' is provided for the simulation tasks. The simini file can be deleted upon request at the beginning of any scenario run, forcing CAPRI to re-create it before the scenario shock is introduced.
+Technically, the calibration of the biofuel demand system and the Armington bilateral trade system is not directly linked to the BASELINE mode, but also executed every time when the simini file is missing (by create_sim_ini_gdx module).