CAPRI Online Manual (update)

User Tools

Site Tools

Trace:
baseline_generation

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
baseline_generation [2020/04/25 06:45] – matszbaseline_generation [2022/11/07 10:23] (current) – external edit 127.0.0.1
Line 12: Line 12:
   * The second example is that of FAPRI model, where a so-called melting down meeting is organised where the modellers responsible for specific parts of the system come together with market experts. Results are discussed, parameters and assumptions changed until there is consensus. Little is known about how the process works exactly, but both examples underline the interaction between model mechanisms and ex-ante expectations of market experts.   * The second example is that of FAPRI model, where a so-called melting down meeting is organised where the modellers responsible for specific parts of the system come together with market experts. Results are discussed, parameters and assumptions changed until there is consensus. Little is known about how the process works exactly, but both examples underline the interaction between model mechanisms and ex-ante expectations of market experts.
  
-As is the case in other agencies, the CAPRI baseline is also fed by external (“expert”) forecasts, as well as by trend forecasts using data from the national ‘COCO’ and regionalized CAPREG databases (Chapters [[The Complete and Consistent Data Base (COCO) for the national scale]] and [[The Regionalised Data Base (CAPREG)]]). The purpose of these trend estimates is, on the one hand, to compare expert forecasts with a purely technical extrapolation of time series and, on the other hand, to provide a ‘safety net’ position in case no values from external projection are available. Usually the projections for a CAPRI baseline are a combination of expert data (e.g. from FAO, European Commission, World Bank, other research teams and even private entreprises) and simple statistical trends of data contained in the CAPRI database. +As is the case in other agencies, the CAPRI baseline is also fed by external (“expert”) forecasts, as well as by trend forecasts using data from the national ‘COCO’ and regionalized CAPREG databases (sections [[the capri data base#The Complete and Consistent Data Base (COCO) for the national scale]] and [[the capri data base#The Regionalised Data Base (CAPREG)]]). The purpose of these trend estimates is, on the one hand, to compare expert forecasts with a purely technical extrapolation of time series and, on the other hand, to provide a ‘safety net’ position in case no values from external projection are available. Usually the projections for a CAPRI baseline are a combination of expert data (e.g. from FAO, European Commission, World Bank, other research teams and even private entreprises) and simple statistical trends of data contained in the CAPRI database. 
  
 =====Overview of CAPRI baseline processes===== =====Overview of CAPRI baseline processes=====
Line 20: Line 20:
 **Figure 10: Overview of CAPRI baseline process** **Figure 10: Overview of CAPRI baseline process**
  
-{{::figure_10.png?600|}} \\ Source: own illustration+{{::figure_10.png?600|Source: own illustration}} 
  
 The forecast tool CAPTRD uses the consolidated national and regional time series from COCO and CAPREG together with external projections from the AgLink model. The result is a projection for the key variables in the agricultural sector (activity levels and market balances) of all regions in the supply models (EU+) that is consistent with the supply model equations.  The forecast tool CAPTRD uses the consolidated national and regional time series from COCO and CAPREG together with external projections from the AgLink model. The result is a projection for the key variables in the agricultural sector (activity levels and market balances) of all regions in the supply models (EU+) that is consistent with the supply model equations. 
  
-  - Next task is the market model calibration. That task uses the same AgLink projections, complemented with the harmonized trade database GLOBAL (see section [[The global database components]]), the baseline policy files, the regional data for the base year (CAPREG) and the regional trends coming from CAPTRD. The output includes a market data set that is consistent with the regional trends, with calibrated parameters to steer behavioural functions, and adds producer prices to be used by the supply models.+  - Next task is the market model calibration. That task uses the same AgLink projections, complemented with the harmonized trade database GLOBAL (see section [[the capri data base#The global database components]]), the baseline policy files, the regional data for the base year (CAPREG) and the regional trends coming from CAPTRD. The output includes a market data set that is consistent with the regional trends, with calibrated parameters to steer behavioural functions, and adds producer prices to be used by the supply models.
   - The third task is the calibration of the supply models. This step also uses the regional data base, regional trends, and policy files, and calibrates various technical and behavioural economic parameters of the supply models so that the projected regional production is the optimal production at the producer prices coming from the market model calibration.   - The third task is the calibration of the supply models. This step also uses the regional data base, regional trends, and policy files, and calibrates various technical and behavioural economic parameters of the supply models so that the projected regional production is the optimal production at the producer prices coming from the market model calibration.
   - Finally, the modeller typically wants to perform a simulation using all the calibrated parameters and projected data. The purpose is twofold: to verify that the calibration of the baseline worked as intended and to generate all reports for inspection in the GUI.    - Finally, the modeller typically wants to perform a simulation using all the calibrated parameters and projected data. The purpose is twofold: to verify that the calibration of the baseline worked as intended and to generate all reports for inspection in the GUI. 
Line 58: Line 58:
   * //Expost//: defined from the length of the series in CAPREG output //res_time_series.gdx//   * //Expost//: defined from the length of the series in CAPREG output //res_time_series.gdx//
   * //Exante//: covering any sequence of intermediate result years up to the user specified final year((For technical reasons some years are “obligatory” result years, for example the year immediately following after the last ex post year.)).    * //Exante//: covering any sequence of intermediate result years up to the user specified final year((For technical reasons some years are “obligatory” result years, for example the year immediately following after the last ex post year.)). 
-  * //ExanteD//: Ex ante years with additional COCO1 data (assigned in ‘//captrd\ load_coco1_data.gms//’)+  * //ExanteD//: Ex ante years with additional COCO1 data (assigned in ‘//captrd/load_coco1_data.gms//’)
   * //ExpostT//: Union of Expost and ExanteD = years with data for trend estimation   * //ExpostT//: Union of Expost and ExanteD = years with data for trend estimation
  
Line 67: Line 67:
 \end{equation} \end{equation}
  
-The weighting with the trend was introduced in the exploration phase based on the following considerations and experience. First of all, it reflects the fact that statistics from the early years (mid eighties) are often less reliable then those from later years. Secondly, even if they are reliable, older data will tend to contribute less useful information than more recent ones due to ongoing structural change. For this reason we have discarded any years before 1992 for the New MS, for example, but the data from the mid 90ies may nonetheless represent a situation of transition that should count less than the recent past. In technical terms the step 1 estimates are found by a grid search over selected values of parameter c with analytical OLS estimates for parameters a and b (see //‘captrd\estimate_trends.gms’//) that have been found identical to those of the econometric package Eviews for a given value of c (holds also for wSSE). +The weighting with the trend was introduced in the exploration phase based on the following considerations and experience. First of all, it reflects the fact that statistics from the early years (mid eighties) are often less reliable then those from later years. Secondly, even if they are reliable, older data will tend to contribute less useful information than more recent ones due to ongoing structural change. For this reason we have discarded any years before 1992 for the New MS, for example, but the data from the mid 90ies may nonetheless represent a situation of transition that should count less than the recent past. In technical terms the step 1 estimates are found by a grid search over selected values of parameter c with analytical OLS estimates for parameters a and b (see //‘captrd/estimate_trends.gms’//) that have been found identical to those of the econometric package Eviews for a given value of c (holds also for wSSE). 
  
 ====Step 2.1: Consistency constraints in the trend projection tool==== ====Step 2.1: Consistency constraints in the trend projection tool====
Line 77: Line 77:
 The confidence interval from the Step 1 trend estimation will not help, as it will be centred around the last projection value and as it will simply be quite large in case of a bad R². However, we may use the idea underlying the usual test statistics for the parameters related to the trend (//a,b,c//). These statistics test the probability of (//a,b,c//) being significantly different from zero. It can be shown that these tests are directly related to R² of the regression. If the zero hypotheses would be true, i.e. if the estimated parameters would have a high probability of being zero, we would not use the trend line, but the mean of the series instead. The confidence interval from the Step 1 trend estimation will not help, as it will be centred around the last projection value and as it will simply be quite large in case of a bad R². However, we may use the idea underlying the usual test statistics for the parameters related to the trend (//a,b,c//). These statistics test the probability of (//a,b,c//) being significantly different from zero. It can be shown that these tests are directly related to R² of the regression. If the zero hypotheses would be true, i.e. if the estimated parameters would have a high probability of being zero, we would not use the trend line, but the mean of the series instead.
  
-This reasoning is the basis for the supports derived from the Step 1 estimates in CAPTRD (//‘captrd\define_stats_and_supports.gms’//), after some modifications. First of all, we used a three-year average based on the last known values as the fallback position and not the mean of the series. Secondly, in typical econometric analysis, test statistics would only be reported for the final estimation layout, some variables would have been dropped from the regression beforehand if certain probability thresholds are undercut. For our applications, we opted for a continuous rule as the choice of threshold values is arbitrary. The smaller the weighted R² the stronger the estimates are drawn towards our H0 – the value is equal to the recent three year average:+This reasoning is the basis for the supports derived from the Step 1 estimates in CAPTRD (//‘captrd/define_stats_and_supports.gms’//), after some modifications. First of all, we used a three-year average based on the last known values as the fallback position and not the mean of the series. Secondly, in typical econometric analysis, test statistics would only be reported for the final estimation layout, some variables would have been dropped from the regression beforehand if certain probability thresholds are undercut. For our applications, we opted for a continuous rule as the choice of threshold values is arbitrary. The smaller the weighted R² the stronger the estimates are drawn towards our H0 – the value is equal to the recent three year average:
  
 \begin{equation} \begin{equation}
Line 100: Line 100:
 How is this rule motivated? If R² for a certain time series is 100%, in other words: for a perfect fit, the restricted trend estimate is fully drawn towards the unrestricted Step 1 estimate. If R² is zero, the trend curve does not explain any of the weighted variance of the series. Consequently, the support is equal to the ‘base data’. The ‘base data’ represent a three-year average around the last three known years. For all cases in between, the supports are the weighted average of the unrestricted trend estimate weighted with R² and the three-year average weighted with (1-R²). Generally, all trend estimates are restricted to the non-negative domain. How is this rule motivated? If R² for a certain time series is 100%, in other words: for a perfect fit, the restricted trend estimate is fully drawn towards the unrestricted Step 1 estimate. If R² is zero, the trend curve does not explain any of the weighted variance of the series. Consequently, the support is equal to the ‘base data’. The ‘base data’ represent a three-year average around the last three known years. For all cases in between, the supports are the weighted average of the unrestricted trend estimate weighted with R² and the three-year average weighted with (1-R²). Generally, all trend estimates are restricted to the non-negative domain.
  
-The above definition of supports works for series with //expost// data from CAPREG only as well as for those series with an extended set of observations (//expostT//, see above). The only difference is whether the three year average denoted above simply with “bas” is calculated using the three last years from set //expost// or from set //expostT// (BASM or BAST in //‘captrd\define_stats_and_supports.gms’//). +The above definition of supports works for series with //expost// data from CAPREG only as well as for those series with an extended set of observations (//expostT//, see above). The only difference is whether the three year average denoted above simply with “bas” is calculated using the three last years from set //expost// or from set //expostT// (BASM or BAST in //‘captrd/define_stats_and_supports.gms’//). 
  
 Our objective function for Step 2 will be the sum of squared deviations from the supports defined above, weighted with the variance of the error terms from the first step: Our objective function for Step 2 will be the sum of squared deviations from the supports defined above, weighted with the variance of the error terms from the first step:
Line 116: Line 116:
 The constraints in the trend projection enforce mutual compatibility between baseline forecasts for individual series in the light of relations between these series, either based on definitions as ‘production equals yield times area’ or on technical relations between series as the balance between energy deliveries from feed use and energy requirements from the animal herds. The set of constraints is deemed to be exhaustive in the sense as any further restriction would either not add information or require data beyond those available. The underlying data set takes into account all agricultural activities and products according to the definition of the Economic Accounts for Agriculture. The constraints in the trend projection enforce mutual compatibility between baseline forecasts for individual series in the light of relations between these series, either based on definitions as ‘production equals yield times area’ or on technical relations between series as the balance between energy deliveries from feed use and energy requirements from the animal herds. The set of constraints is deemed to be exhaustive in the sense as any further restriction would either not add information or require data beyond those available. The underlying data set takes into account all agricultural activities and products according to the definition of the Economic Accounts for Agriculture.
  
-The constraints discussed in the following (from //‘captrd\equations.gms’//) can be seen as a minimum set of consistency conditions necessary for a projection of agricultural variables. The full projection tool features further constraints especially relating to price feedbacks on supply and demand.+The constraints discussed in the following (from //‘captrd/equations.gms’//) can be seen as a minimum set of consistency conditions necessary for a projection of agricultural variables. The full projection tool features further constraints especially relating to price feedbacks on supply and demand.
  
 ===Constraints relating to market balances and yields=== ===Constraints relating to market balances and yields===
  
-Closed market balances (CAPTRD eq. MBAL_) define the first set of constraints and state that the sum of imports (IMPT) and production (GROF) must be equal to the sum of feed (FEDM) and seed (SEDM) use, human consumption (HCOM), processing (INDM,PRCM,BIOF), losses (LOSM) and exports (EXPT):+Closed market balances (CAPTRD eq. MBAL_ ) define the first set of constraints and state that the sum of imports (IMPT) and production (GROF) must be equal to the sum of feed (FEDM) and seed (SEDM) use, human consumption (HCOM), processing (INDM,PRCM,BIOF), losses (LOSM) and exports (EXPT):
  
 \begin{align} \begin{align}
Line 334: Line 334:
 ===Constraints relating to growth rates=== ===Constraints relating to growth rates===
  
-During estimation, a number of safeguards regarding the size of the implicit growth rates had been introduced in the course of various past CAPRI projects (bounds mainly found in //‘captrd\fix_est.gms’//):+During estimation, a number of safeguards regarding the size of the implicit growth rates had been introduced in the course of various past CAPRI projects (bounds mainly found in //‘captrd/fix_est.gms’//):
  
   * In general, input or output coefficients (yields) are not allowed to change by more than +/- 2.5 % per annum, with a higher ranges for feed input coefficients (+/- 10 % and +/  5 % for non-marketable fodder).   * In general, input or output coefficients (yields) are not allowed to change by more than +/- 2.5 % per annum, with a higher ranges for feed input coefficients (+/- 10 % and +/  5 % for non-marketable fodder).
Line 340: Line 340:
   * The number of young cows (or sows) needed for replacement may only change up to +/  20 % around the base period value until the last projection year.   * The number of young cows (or sows) needed for replacement may only change up to +/  20 % around the base period value until the last projection year.
   * Final fattening weights must fall into a corridor of +/- 20% around the base period value.   * Final fattening weights must fall into a corridor of +/- 20% around the base period value.
-  * Milk yields are assumed to increase at least by 0.25% and at most by 1.25% near the EU average with some correction for below or above average initial yields (in //‘captrd\comibounds.gms’//). +  * Milk yields are assumed to increase at least by 0.25% and at most by 1.25% near the EU average with some correction for below or above average initial yields (in //‘captrd/comibounds.gms’//). 
   * Crop yields (except those of very hererogeneous crops like “other fruits” or “other fodder on arable land) should have a minimum yield growth of 0.5%.   * Crop yields (except those of very hererogeneous crops like “other fruits” or “other fodder on arable land) should have a minimum yield growth of 0.5%.
   * Specific (and quite generous) upper limits are applied to prevent unrealistic crop yields (for example: 15 tons/ha for cereals)   * Specific (and quite generous) upper limits are applied to prevent unrealistic crop yields (for example: 15 tons/ha for cereals)
Line 360: Line 360:
   * Total labour must not deviate by more than 5% from forecasts based on coefficients estimated in an earlier study (“CAPRI-DYNASPAT”).    * Total labour must not deviate by more than 5% from forecasts based on coefficients estimated in an earlier study (“CAPRI-DYNASPAT”). 
   * Changes in human consumption per caput for each of the products cannot exceed a growth rate of +/- 2% per annum. Due to some strong and rather implausible trends for total meat and total cereals consumption, the growth rate was restricted to +/- 0.8 % per annum for meat and +/- 0.4% per annum for cereals assuming that trend shifts between single items are more likely than strong trends in aggregate food groups.   * Changes in human consumption per caput for each of the products cannot exceed a growth rate of +/- 2% per annum. Due to some strong and rather implausible trends for total meat and total cereals consumption, the growth rate was restricted to +/- 0.8 % per annum for meat and +/- 0.4% per annum for cereals assuming that trend shifts between single items are more likely than strong trends in aggregate food groups.
-  * A downward sloping corridor is defined for subsistence consumption of raw milk (in ‘captrd\comibounds.gms’).+  * A downward sloping corridor is defined for subsistence consumption of raw milk (in ‘captrd/comibounds.gms’).
   * Changes in prices are not allowed to exceed a growth rate of +/- 2% per annum, usually.   * Changes in prices are not allowed to exceed a growth rate of +/- 2% per annum, usually.
-  * Expert supports for biofuel related variables are given high priority with mostly tight corridors around these supports (in //‘captrd\biobounds.gms’//).+  * Expert supports for biofuel related variables are given high priority with mostly tight corridors around these supports (in //‘captrd/biobounds.gms’//).
   * If a variable has dropped to zero according to recent COCO data it will be fixed to zero.    * If a variable has dropped to zero according to recent COCO data it will be fixed to zero. 
  
Line 369: Line 369:
 The definition of expert “supports” allows for provision of a mean and a standard deviation for all elements, and it is particularly useful for items for which the AgLink forcasts in step 3 are missing, or where there are other reasons for stability problems, such as missing historical data or very short time series  The definition of expert “supports” allows for provision of a mean and a standard deviation for all elements, and it is particularly useful for items for which the AgLink forcasts in step 3 are missing, or where there are other reasons for stability problems, such as missing historical data or very short time series 
  
-The expert supports are dealt with in //’captrd\expert_support.gms’//. Currently, mainly three sources can be distinguished:+The expert supports are dealt with in //’captrd/expert_support.gms’//. Currently, mainly three sources can be distinguished:
  
   * Support for the development of the sugar and sugar beet sectors, evolved from a small study with the seed production company KWS   * Support for the development of the sugar and sugar beet sectors, evolved from a small study with the seed production company KWS
Line 459: Line 459:
 In Step 3, results from external projections on market balance positions (production, consumption, net trade etc.) and on activity levels for EU aggregates (EU15, EU12) are added. Currently, these projections are provided by Aglink-COSIMO model projections. The baseline of Aglink-COSIMO integrates the market outlook results from DG-AGRI, but is also globally harmonised, so that it also enters the baseline generation for the market model of CAPRI. In Step 3, results from external projections on market balance positions (production, consumption, net trade etc.) and on activity levels for EU aggregates (EU15, EU12) are added. Currently, these projections are provided by Aglink-COSIMO model projections. The baseline of Aglink-COSIMO integrates the market outlook results from DG-AGRI, but is also globally harmonised, so that it also enters the baseline generation for the market model of CAPRI.
  
-Integration of results from another modelling system is a challenging exercise as neither data nor definitions of products and market balance positions are fully harmonized. That holds especially for Aglink-COSIMO, where at least in the past the mnemonics had even not been harmonized across equations of the model itself. After a restructuring exercise in 2010, that had somewhat been improved. The ingredients in the mapping process are first a list of the codes for the regions, products and items used in Aglink-COSIMO (‘//baseline\aglink*_sets.gms//’, where * can be 2009 or 2010 to differentiate the versions before and after the restructuring). A second program, (//‘baseline\aglink*_mappings.gms//’) links the CAPRI regions, products and items to the mnemonics and Aglink-COSIMO, and a larger program (‘//baseline\loag_aglink*.gms//’) then uses the mapping to assign them to the CAPRI code world.+Integration of results from another modelling system is a challenging exercise as neither data nor definitions of products and market balance positions are fully harmonized. That holds especially for Aglink-COSIMO, where at least in the past the mnemonics had even not been harmonized across equations of the model itself. After a restructuring exercise in 2010, that had somewhat been improved. The ingredients in the mapping process are first a list of the codes for the regions, products and items used in Aglink-COSIMO (‘//baseline/aglink*_sets.gms//’, where * can be 2009 or 2010 to differentiate the versions before and after the restructuring). A second program, (//‘baseline/aglink*_mappings.gms//’) links the CAPRI regions, products and items to the mnemonics and Aglink-COSIMO, and a larger program (‘//baseline/loag_aglink*.gms//’) then uses the mapping to assign them to the CAPRI code world.
    
 Aglink-COSIMO currently features results at EU15 and EU12 level. It is hence not possible to funnel the Aglink-COSIMO results into Step 2 above without an assumption of the share of the individual Member States.   Aglink-COSIMO currently features results at EU15 and EU12 level. It is hence not possible to funnel the Aglink-COSIMO results into Step 2 above without an assumption of the share of the individual Member States.  
Line 483: Line 483:
 The Aglink-COSIMO projections currently run to 2020 or a few years beyond. For climate related applications CAPRI has to tackle projections up to 2030 or even 2050. CAPRI projections up to 2030 have been prepared in the context of EC4MACS project ([[http://www.ec4macs.eu]]). The methodology was quite simple: The year 2020 projection (usually prepared in the same run of CAPTRD) has been extrapolated in a nonlinear dampened (logistic) fashion (in //‘define_eu_supports.gms’//) with some additional bounds to prevent unreasonable increases of certain variables (nonnegativity already provided a good lower bound). Together with the information in the time series database this has been an ad hoc but operational procedure to address the 2030 horizon, but it would have been inappropriate for a move to the long run up to 2050 as required for a recent study on behalf of DG CLIMA((Service contract on “Model based assessment of EU energy and climate change policies for post-2012 regime” (Tender DG ENV.C.5/SER/2009/0036), coordinated by the Energy-Economy-Environment Modelling Laboratory (E3MLab), National Technical University of Athens with the International Institute for Applied Systems Analysis (IIASA) and EuroCARE as subcontractors.)).  The Aglink-COSIMO projections currently run to 2020 or a few years beyond. For climate related applications CAPRI has to tackle projections up to 2030 or even 2050. CAPRI projections up to 2030 have been prepared in the context of EC4MACS project ([[http://www.ec4macs.eu]]). The methodology was quite simple: The year 2020 projection (usually prepared in the same run of CAPTRD) has been extrapolated in a nonlinear dampened (logistic) fashion (in //‘define_eu_supports.gms’//) with some additional bounds to prevent unreasonable increases of certain variables (nonnegativity already provided a good lower bound). Together with the information in the time series database this has been an ad hoc but operational procedure to address the 2030 horizon, but it would have been inappropriate for a move to the long run up to 2050 as required for a recent study on behalf of DG CLIMA((Service contract on “Model based assessment of EU energy and climate change policies for post-2012 regime” (Tender DG ENV.C.5/SER/2009/0036), coordinated by the Energy-Economy-Environment Modelling Laboratory (E3MLab), National Technical University of Athens with the International Institute for Applied Systems Analysis (IIASA) and EuroCARE as subcontractors.)). 
  
-For the long run evolution of food production a link has been established to long run projections from two major agencies (FAO 2006 and the IMPACT projections in Rosegrant et al 2009, see also Rosegrant et al 2008). This linkage required mappings to bridge differences in definitions (see //‘gams\global\f2050_impact.gms’// called when running //‘gams\global.gms’//).+For the long run evolution of food production a link has been established to long run projections from two major agencies (FAO 2006 and the IMPACT projections in Rosegrant et al 2009, see also Rosegrant et al 2008). This linkage required mappings to bridge differences in definitions (see //‘gams/global/f2050_impact.gms’// called when running //‘gams/global.gms’//).
  
-Furthermore, methodology was needed to avoid a break in the projections at the transition of medium run expert information (Aglink-COSIMO, up to 2020) and long run information (FAO/IFPRI for 2050). For this purpose a variable weighting scheme is introduced (in //‘gams\captrd\expert_support.gms’//) that gives an increasing weight to our “long run” sources (FAO/IFPRI) as the projection horizon approaches 2050. This tends to give projections that gradually approach the long run sources, for example as in the case of pork production in Hungary (taken from a baseline established in November 2011).+Furthermore, methodology was needed to avoid a break in the projections at the transition of medium run expert information (Aglink-COSIMO, up to 2020) and long run information (FAO/IFPRI for 2050). For this purpose a variable weighting scheme is introduced (in //‘gams/captrd/expert_support.gms’//) that gives an increasing weight to our “long run” sources (FAO/IFPRI) as the projection horizon approaches 2050. This tends to give projections that gradually approach the long run sources, for example as in the case of pork production in Hungary (taken from a baseline established in November 2011).
  
 **Figure 11: Pork production in Hungary as an example for merging medium run and long run a priori information in the CAPRI baseline approach** **Figure 11: Pork production in Hungary as an example for merging medium run and long run a priori information in the CAPRI baseline approach**
  
-{{::figure11.png?600|}} \\ Source: own elaboration+{{::figure_11.png?600|Source: own elaboration}} 
  
 The example has been chosen because historical trends (and Aglink-COSIMO projections) on the one hand and long run expectations differ markedly. This is not unusual because medium run forecasts often give a stronger weight to recent production trends, often indicating a stagnating or declining production in the EU, whereas the long run studies tend to focus on the global growth of food demand in the coming decades. The simple trends (filled triangles) would evidently give unreasonable, even negative forecasts after 2030. Already the imposition of constraints from relationships to other series would stabilise the projections and imply some recovery after 2030 (filled squares). The year 2020 supports from Aglink-COSIMO (not shown) produces some upward correction of the step 2 results for 2020, giving a final projection (filled circles) of about 375 ktons for pork production in Hungary. This is also the starting point for the specification of the long run support (empty circles) which is a weighted average of two components. The first is a linear interpolation to the external projection from FAO/IFPRI for 2050 (empty triangles).  The second is a nonlinear damped extrapolation of the medium run projection beyond 2020 (empty squares). Changing the weight for the first component (FAO/IFPRI support) with increasing projection horizon creates a long run target value (empty circles) that gives a smooth transition from the medium to the long run. As the final projections (filled circles) tend to follow these target values, they show a turning point in the future evolution of pork production in Hungary that ultimately reflects the consideration of increasing global demand underlying the FAO/IFPRI projections.  The example has been chosen because historical trends (and Aglink-COSIMO projections) on the one hand and long run expectations differ markedly. This is not unusual because medium run forecasts often give a stronger weight to recent production trends, often indicating a stagnating or declining production in the EU, whereas the long run studies tend to focus on the global growth of food demand in the coming decades. The simple trends (filled triangles) would evidently give unreasonable, even negative forecasts after 2030. Already the imposition of constraints from relationships to other series would stabilise the projections and imply some recovery after 2030 (filled squares). The year 2020 supports from Aglink-COSIMO (not shown) produces some upward correction of the step 2 results for 2020, giving a final projection (filled circles) of about 375 ktons for pork production in Hungary. This is also the starting point for the specification of the long run support (empty circles) which is a weighted average of two components. The first is a linear interpolation to the external projection from FAO/IFPRI for 2050 (empty triangles).  The second is a nonlinear damped extrapolation of the medium run projection beyond 2020 (empty squares). Changing the weight for the first component (FAO/IFPRI support) with increasing projection horizon creates a long run target value (empty circles) that gives a smooth transition from the medium to the long run. As the final projections (filled circles) tend to follow these target values, they show a turning point in the future evolution of pork production in Hungary that ultimately reflects the consideration of increasing global demand underlying the FAO/IFPRI projections. 
Line 527: Line 527:
 In order to keep developments at regional and national level comparable, relative changes in activity levels are not allowed to deviate very far from the national development. These bounds are widened in cases of infeasibilities. In order to keep developments at regional and national level comparable, relative changes in activity levels are not allowed to deviate very far from the national development. These bounds are widened in cases of infeasibilities.
  
-Table below contains an example of the final output of the trends estimation task (C:\....CAPRI\STAR\star_2.4\output\results\baseline\results_BBYY.gdx), where BB stands for base year and YY for simulation year). Its main purpose is to provide with explanations on the variables of this output and, thus, a possibility to review the results in a step-by-step manner.+Table below contains an example of the final output of the trends estimation task (C:/....CAPRI/STAR/star_2.4/output/results/baseline/results_BBYY.gdx), where BB stands for base year and YY for simulation year). Its main purpose is to provide with explanations on the variables of this output and, thus, a possibility to review the results in a step-by-step manner.
  
 **Table 24: Example of the final output of the trends estimation task and description of the variables** **Table 24: Example of the final output of the trends estimation task and description of the variables**
Line 559: Line 559:
 After the Task on Trends generation have been successfully completed, meaning that the projections for the defined (in GUI or a batch file) future years (currently, 2015, 2020, 2025 and 2030 are available) have been produced, the next step in the Baseline generation process ("Generate baseline" workstep in CAPRI GUI) is to calibrate the CAPRI global trade model. In the CAPRI GUI this refers to the task "Baseline calibration of market model". After the Task on Trends generation have been successfully completed, meaning that the projections for the defined (in GUI or a batch file) future years (currently, 2015, 2020, 2025 and 2030 are available) have been produced, the next step in the Baseline generation process ("Generate baseline" workstep in CAPRI GUI) is to calibrate the CAPRI global trade model. In the CAPRI GUI this refers to the task "Baseline calibration of market model".
  
-The calibration of the market model is steered by the C:\...\CAPRI\gams\capmod.gms file. The relevant parts of the code are activated by setting the setglobal 'BASELINE' to ON.+The calibration of the market model is steered by the C:/.../CAPRI/gams/capmod.gms file. The relevant parts of the code are activated by setting the setglobal 'BASELINE' to ON.
  
 ====Stage I: Data preparation and balancing==== ====Stage I: Data preparation and balancing====
Line 577: Line 577:
 ===Data preparation=== ===Data preparation===
  
-Before actually performing the calibration of the market model parameters, CAPRI first loads the necessary sets, parameters and data. These refer to periods (years), regions, activities, commodities, agricultural policies (e.g., premiums, quotas, rural development payments, set-aside requirements), environmental indicators, feed and fertilizer requirements, nutrient content of the commodities, global warming potentials, and other necessary input. The data loaded also includes two very important for this calibration step files: C:\...\CAPRI\output\results\capreg\res_BBCC.gdx and C:\...\CAPRI\output\results\baseline\trends_BBYY.gdx. The first file, res_BBCC.gdx, includes the results of generation of data for the base year (BB, currently 2012) for European countries and Turkey (CC) at NUTS0, NUTS1 and NUTS2 aggregation levels (GUI workstep "Build database", task "Build regional database"). The second file, trends_BBYY.gdx, includes the results of trends generation task (see sections above) for all of the European countries and Turkey at NUTS0, NUTS1 and NUTS2 aggregation levels for the target simulation year (currently, 2030).+Before actually performing the calibration of the market model parameters, CAPRI first loads the necessary sets, parameters and data. These refer to periods (years), regions, activities, commodities, agricultural policies (e.g., premiums, quotas, rural development payments, set-aside requirements), environmental indicators, feed and fertilizer requirements, nutrient content of the commodities, global warming potentials, and other necessary input. The data loaded also includes two very important for this calibration step files: C:/.../CAPRI/output/results/capreg/res_BBCC.gdx and C:/.../CAPRI/output/results/baseline/trends_BBYY.gdx. The first file, res_BBCC.gdx, includes the results of generation of data for the base year (BB, currently 2012) for European countries and Turkey (CC) at NUTS0, NUTS1 and NUTS2 aggregation levels (GUI workstep "Build database", task "Build regional database"). The second file, trends_BBYY.gdx, includes the results of trends generation task (see sections above) for all of the European countries and Turkey at NUTS0, NUTS1 and NUTS2 aggregation levels for the target simulation year (currently, 2030).
  
 Constraints, requirements, policies and other data loaded including base year and trends data (i.e., of res_BBCC.gdx  and trends_BBYY.gdx files) are subject to certain (mainly non-major) adjustments, additional calculations and assumptions that serve the purposes of data balancing, checking and provision of necessary for the calibration information. These include, for example, deleting positions not needed during the calibration run, (re-)assigning parameter names, deleting tiny quantities, checks for production without activity levels, possible empty projections and negative inland waters, setting the output coefficients for young animals equal to the ones at country (EU MSs, as young animals are not presented in the non-EU countries) level if missing at regional level, correcting fat and protein content of raw milk, assumption that second generation biofuels are produced 50/50 by agricultural residuals and new energy crops, etc. Constraints, requirements, policies and other data loaded including base year and trends data (i.e., of res_BBCC.gdx  and trends_BBYY.gdx files) are subject to certain (mainly non-major) adjustments, additional calculations and assumptions that serve the purposes of data balancing, checking and provision of necessary for the calibration information. These include, for example, deleting positions not needed during the calibration run, (re-)assigning parameter names, deleting tiny quantities, checks for production without activity levels, possible empty projections and negative inland waters, setting the output coefficients for young animals equal to the ones at country (EU MSs, as young animals are not presented in the non-EU countries) level if missing at regional level, correcting fat and protein content of raw milk, assumption that second generation biofuels are produced 50/50 by agricultural residuals and new energy crops, etc.
  
-Next, FAO data on the non-European countries as well as the trade flows among all of the countries (country trade blocks) accounted for in CAPRI are loaded. These FAO data together with the European data, which has already been subjected to certain adjustments as described in the previous paragraph, undergo the, so-called, data preparation step. This process is controlled by C:\...\CAPRI\gams\arm\market1.gms file which calls the C:\...\CAPRI\gams\arm\data_prep.gms file - specifically for this step. The data preparation step mostly refers to the base year and includes: among else, modification of GDP to fit the sum of final household expenditure, final government expenditure, gross capital formation and current account balance; import and export flows to be in line with net trade from production minus demand; scaling of demand side to fit production plus net trade; estimation of consumer prices for some countries, if missing; calculation of nutrient consumption per head and day as net of losses in distribution and households; scaling of outliers in prices etc. This step as well provides with estimation of yearly change factors beyond the base year: for prices, GDP, population, quantities and areas. Additionally, i) substitution elasticities (i.e., p_rhoX, where X indicates continuation of the parameter name) for bio-fuel feedstocks, feed, dairy products, sugar, table grapes, tobacco, cheese, fresh milk products, fruits, vegetables, distilled dried grains and rice for the CAPRI demand system((See section [[Market module for agricultural outputs#Overview on the market model]] on Overview of the market model "CAPRI comprises a two stage Armington system: on the top level, the composition of total demand from imports and domestic sales is determined, as a function of the relation between the domestic price and the average import price. The lower stage determines the import shares from different origins and defines the average import price.")), and ii) transformation elasticity for oil seed processing and land supply elasticities are assigned.+Next, FAO data on the non-European countries as well as the trade flows among all of the countries (country trade blocks) accounted for in CAPRI are loaded. These FAO data together with the European data, which has already been subjected to certain adjustments as described in the previous paragraph, undergo the, so-called, data preparation step. This process is controlled by C:/.../CAPRI/gams/arm/market1.gms file which calls the C:/.../CAPRI/gams/arm/data_prep.gms file - specifically for this step. The data preparation step mostly refers to the base year and includes: among else, modification of GDP to fit the sum of final household expenditure, final government expenditure, gross capital formation and current account balance; import and export flows to be in line with net trade from production minus demand; scaling of demand side to fit production plus net trade; estimation of consumer prices for some countries, if missing; calculation of nutrient consumption per head and day as net of losses in distribution and households; scaling of outliers in prices etc. This step as well provides with estimation of yearly change factors beyond the base year: for prices, GDP, population, quantities and areas. Additionally, i) substitution elasticities (i.e., p_rhoX, where X indicates continuation of the parameter name) for bio-fuel feedstocks, feed, dairy products, sugar, table grapes, tobacco, cheese, fresh milk products, fruits, vegetables, distilled dried grains and rice for the CAPRI demand system((See section [[scenario simulation#Overview on the market model]] on Overview of the market model "CAPRI comprises a two stage Armington system: on the top level, the composition of total demand from imports and domestic sales is determined, as a function of the relation between the domestic price and the average import price. The lower stage determines the import shares from different origins and defines the average import price.")), and ii) transformation elasticity for oil seed processing and land supply elasticities are assigned.
  
-Together with the data, equations of the CAPRI market module are loaded. They are described in detail in section [[Market module for agricultural outputs]]. These equations include behavioural functions for market demand including expenditure function, feed demand, blocks for dairy products, oilseeds processing and biofuels, netput functions, trade equations and balances, equations for prices and price transmission, functions for trade policies and for intervention stocks. There are additionally two crucial for data calibration functions: minimization of deviation of estimated values from the observed data. These two functions are described in detail later in this section.    +Together with the data, equations of the CAPRI market module are loaded. They are described in detail in section [[scenario simulation#Market module for agricultural outputs]]. These equations include behavioural functions for market demand including expenditure function, feed demand, blocks for dairy products, oilseeds processing and biofuels, netput functions, trade equations and balances, equations for prices and price transmission, functions for trade policies and for intervention stocks. There are additionally two crucial for data calibration functions: minimization of deviation of estimated values from the observed data. These two functions are described in detail later in this section.    
  
 ===Data balancing=== ===Data balancing===
  
-After data preparation, data calibration for the base (currently, 2012) and simulation years (currently, 2030) take place. The main file steering the data balancing process is C:\...\CAPRI\gams\arm\data_cal.gms, which in turned is included in arm\market1.gms. +After data preparation, data calibration for the base (currently, 2012) and simulation years (currently, 2030) take place. The main file steering the data balancing process is C:/.../CAPRI/gams/arm/data_cal.gms, which in turn is included in arm/market1.gms. 
  
 //Data balancing for the base year// \\ //Data balancing for the base year// \\
Line 593: Line 593:
 Data calibration for the base year aims at modifying the base year data to fit the system of equations of the market module. Some of the parameters defined in Stage I (e.g., p_rhoX) as well as parameter values and bounds defined at this stage are used. For example, starting points and corridors for quantity variables are set (e.g., calculating of world production to define correction corridor for calibration of production/demand/trade flows globally), global TRQ data are converted into ad valorem tariffs and checked for consistency and completeness, policy variables for the EU market model such as e.g., intervention stocks, are loaded. Also, starting values for prices of dairy products are estimated. In particular, a non-linear programming model is used, where the objective function is formulated as a Highest Posterior Density function. The value of this objective function equals sum of squared deviations of fat and protein prices, fat and protein content of milk products and processing margins of milk products from the respective means, weighted with the a priori variances. The means are defined as parameters based on the prices and fat and protein content of milk in the base year. The objective function is restricted by the balance: fat and protein of raw milk delivered to dairies shall equal fat and protein content of dairy products. The model is solved by minimizing the value of highest posterior density, hence minimizing the differences between the variables and their means. Prices of milk products are then defined as: product of fat and protein content and of far and protein prices plus processing margin. Furthermore, administrative prices for cereals and dairy products, and minimal import prices for cereals are constructed. Data calibration for the base year aims at modifying the base year data to fit the system of equations of the market module. Some of the parameters defined in Stage I (e.g., p_rhoX) as well as parameter values and bounds defined at this stage are used. For example, starting points and corridors for quantity variables are set (e.g., calculating of world production to define correction corridor for calibration of production/demand/trade flows globally), global TRQ data are converted into ad valorem tariffs and checked for consistency and completeness, policy variables for the EU market model such as e.g., intervention stocks, are loaded. Also, starting values for prices of dairy products are estimated. In particular, a non-linear programming model is used, where the objective function is formulated as a Highest Posterior Density function. The value of this objective function equals sum of squared deviations of fat and protein prices, fat and protein content of milk products and processing margins of milk products from the respective means, weighted with the a priori variances. The means are defined as parameters based on the prices and fat and protein content of milk in the base year. The objective function is restricted by the balance: fat and protein of raw milk delivered to dairies shall equal fat and protein content of dairy products. The model is solved by minimizing the value of highest posterior density, hence minimizing the differences between the variables and their means. Prices of milk products are then defined as: product of fat and protein content and of far and protein prices plus processing margin. Furthermore, administrative prices for cereals and dairy products, and minimal import prices for cereals are constructed.
  
-With the file C:\...\CAPRI\gams\arm\cal_models.gms, the so-called, models, used in calibration of data base are defined and solved. These models represent collection of equations, solutions of which provide with parameter values used for data calibration. The first model (MODEL m_trimSubsExports) calibrates the parameters of the function which defines the values of subsidized exports with and without the increase of market price above the administrative price. The second model (MODEL m_trimInterv) defines parameters of equations for intervention stock changes. It includes an objective function defined as a sum of: squared scaled difference of estimated and observed intervention stock changes and squared scaled parameters for behavioural function of intervention stock changes. This objective function is minimized subject to constraints represented by equations for intervention sales, probability for an undercut of administrative price, release from intervention stock, intervention stock changes and value of the intervention stock. The constraints are equations of the market model (see section [[Market module for agricultural outputs]]).+With the file C:/.../CAPRI/gams/arm/cal_models.gms, the so-called, models, used in calibration of data base are defined and solved. These models represent collection of equations, solutions of which provide with parameter values used for data calibration. The first model (MODEL m_trimSubsExports) calibrates the parameters of the function which defines the values of subsidized exports with and without the increase of market price above the administrative price. The second model (MODEL m_trimInterv) defines parameters of equations for intervention stock changes. It includes an objective function defined as a sum of: squared scaled difference of estimated and observed intervention stock changes and squared scaled parameters for behavioural function of intervention stock changes. This objective function is minimized subject to constraints represented by equations for intervention sales, probability for an undercut of administrative price, release from intervention stock, intervention stock changes and value of the intervention stock. The constraints are equations of the market model (see section [[scenario simulation#Market module for agricultural outputs]]).
  
 The model that calibrates base year data (MODEL m_calMarketBas) is defined in cal_models.gms file as well and includes almost all equations of the market model. In particular: equations for processing margin for dairy products (ProcMargM_), fat and protein balance between raw milk and dairy products (FatsProtBal_), processing margin for oilseeds ProcMargO_, processing yields of oilseeds (procYield_), 1st generation output of biofuels (prodBiof_) and total output of biofuels (MaprBiof_); __balancing and adding up equations__: equations which add production, processing demand, human consumption, feed demand quantities and quantities for processing from single countries (or block of countries) to trade blocks (ProdA_, ProcA_, HconA_, FeedUseA_, Proca_), adding up inside of the Armginton aggregate (total domestic consumption) (ArmBal1_), supply balance (SupBalM_) and imports and exports added up to bilateral trade flows (excluding diagonal element) (impQuant_); __price equations__: 1st stage Armington quantity aggregate (ArmFit1_), 2nd stage Armington quantity aggregate (ArmFit2_), import price relation to producer price (impPrice_), consumer price as average of domestic and import prices (arm1Price_), average price as average of different import prices (arm2Price_), average import price (arm2Val_), consumer price (Cpri_), producer price (PPri_), market price (PMrk_), average market price (MarketPriceAgg_); __trade and tariff equations__: aggregated trade flows (TradeFlowsAgg_), average transportation costs (TransportCostsAgg_), sum of imports under a non-allocated TRQ (TRQImports_), share of the tariff applied for the EU entry price system (EntryPriceDriver_), tariff specific entry price (tarSpecIfEntryPrice_), Cif price (cifPrice_), equation for defining levy (replaces tariff) in case of minimal border prices (FlexLevyNotCut_), cuting flexible levy by specific tariff if it exceeds the bound rate (FlexLevy_), tariffs under bi-lateral TRQs (trqSigmoidFunc_), specific tariffs as function of import quantities, if TRQ is present (tarSpec_, prefTriggerPrice_), tariffs under globally open (not bilaterally allocated) TRQs (tarSpecW_), ad valorem tariffs, if TRQ is present (tarAdval_), ad valorem tariff under not bilaterally allocated TRQs (tarAdValW_), export quantities from bi-lateral trade flows (expQuant_), exports included in the calculation of the export unit values excluding flows under double-zero agreements (nonDoubleZeroExports_), unit value exports (unitValueExports_, valSubsExports_), subsidised export values (EXPs_); __equations for intervention stocks__: probability weight for an undercut of administrative price (probMarketPriceUnderSafetyNet_), intervention sales (buyingToIntervStock_), intervention stock end size (intervStockLevel_), intervention stock changes (intervStockChange_), release from intervention stocks (releaseFromIntervStock_), aggregators for intervention purchases; equation for world market price (wldPrice_), and equation for minimization of deviation from given base year data and estimated data (NSSQ_). The model is solved by minimizing the SSQ value of NSSQ equation which is constrained by all of the rest of the equations included in the model. The model that calibrates base year data (MODEL m_calMarketBas) is defined in cal_models.gms file as well and includes almost all equations of the market model. In particular: equations for processing margin for dairy products (ProcMargM_), fat and protein balance between raw milk and dairy products (FatsProtBal_), processing margin for oilseeds ProcMargO_, processing yields of oilseeds (procYield_), 1st generation output of biofuels (prodBiof_) and total output of biofuels (MaprBiof_); __balancing and adding up equations__: equations which add production, processing demand, human consumption, feed demand quantities and quantities for processing from single countries (or block of countries) to trade blocks (ProdA_, ProcA_, HconA_, FeedUseA_, Proca_), adding up inside of the Armginton aggregate (total domestic consumption) (ArmBal1_), supply balance (SupBalM_) and imports and exports added up to bilateral trade flows (excluding diagonal element) (impQuant_); __price equations__: 1st stage Armington quantity aggregate (ArmFit1_), 2nd stage Armington quantity aggregate (ArmFit2_), import price relation to producer price (impPrice_), consumer price as average of domestic and import prices (arm1Price_), average price as average of different import prices (arm2Price_), average import price (arm2Val_), consumer price (Cpri_), producer price (PPri_), market price (PMrk_), average market price (MarketPriceAgg_); __trade and tariff equations__: aggregated trade flows (TradeFlowsAgg_), average transportation costs (TransportCostsAgg_), sum of imports under a non-allocated TRQ (TRQImports_), share of the tariff applied for the EU entry price system (EntryPriceDriver_), tariff specific entry price (tarSpecIfEntryPrice_), Cif price (cifPrice_), equation for defining levy (replaces tariff) in case of minimal border prices (FlexLevyNotCut_), cuting flexible levy by specific tariff if it exceeds the bound rate (FlexLevy_), tariffs under bi-lateral TRQs (trqSigmoidFunc_), specific tariffs as function of import quantities, if TRQ is present (tarSpec_, prefTriggerPrice_), tariffs under globally open (not bilaterally allocated) TRQs (tarSpecW_), ad valorem tariffs, if TRQ is present (tarAdval_), ad valorem tariff under not bilaterally allocated TRQs (tarAdValW_), export quantities from bi-lateral trade flows (expQuant_), exports included in the calculation of the export unit values excluding flows under double-zero agreements (nonDoubleZeroExports_), unit value exports (unitValueExports_, valSubsExports_), subsidised export values (EXPs_); __equations for intervention stocks__: probability weight for an undercut of administrative price (probMarketPriceUnderSafetyNet_), intervention sales (buyingToIntervStock_), intervention stock end size (intervStockLevel_), intervention stock changes (intervStockChange_), release from intervention stocks (releaseFromIntervStock_), aggregators for intervention purchases; equation for world market price (wldPrice_), and equation for minimization of deviation from given base year data and estimated data (NSSQ_). The model is solved by minimizing the SSQ value of NSSQ equation which is constrained by all of the rest of the equations included in the model.
Line 599: Line 599:
 The NSSQ equation is crucial to the data calibration as it, in its essence, minimizes the difference between the estimated and the observed (already adjusted at the previous stage) data of the base year. Its logic is analogues to the one of equation below: The NSSQ equation is crucial to the data calibration as it, in its essence, minimizes the difference between the estimated and the observed (already adjusted at the previous stage) data of the base year. Its logic is analogues to the one of equation below:
  
-FIXME im dokument nicht nummeriert 
 \begin{equation} \begin{equation}
 SSQ\cdot \sum_{RMS} \sum_{XXX} p\_weight_{RMS}^i=\sum_{RMS} \sum_{XXX} \left( \frac{v_{RMS,XXX}^i-DATA_{RSM,XXX,BAS}^i}{max(DATA_{RSM,XXX,BAS}^i,0.1) \cdot p\_weight_{RMS}^i} \right)^2 SSQ\cdot \sum_{RMS} \sum_{XXX} p\_weight_{RMS}^i=\sum_{RMS} \sum_{XXX} \left( \frac{v_{RMS,XXX}^i-DATA_{RSM,XXX,BAS}^i}{max(DATA_{RSM,XXX,BAS}^i,0.1) \cdot p\_weight_{RMS}^i} \right)^2
 \end{equation} \end{equation}
  
-where SSQ is an artificial variable to be minimized, indices RMS, XXX, BAS and i indicate, respectively, regions, commodities, base year and activities (e.g., production, processing, imports etc.), and p_weight is a parameter of weights between 1 and 100 assigned to regions and activities. These weights are necessary to achieve plausible calibrated values and their specification is the outcome of a trial and error process, inspecting results from data calibration and retrying. They depend on the results of global database and trends generation. On the right hand-side of the equation v stands for a variable to be estimated and DATA – for base year data already adjusted at the data preparation and balancing stage. Hence with this equation squared sum over regions and commodities of differences between estimated and observed values (and or quantities), these differences being scaled by the observed data times the weight parameters, is minimized. Respectively, calibrated base year data fits the system of the market equations, given certain parameter values, and resembles the observed data as closely as possible. The activities implied under the I index include quantities of production, human consumption, feed, processing, processed to biofuels, import and export, producer, consumer and market prices, difference between market prices and import prices to reduce differences between physical and Armington aggregation, consolidated gap between producer and market prices, processing margin, trade flows and transport costs.+where SSQ is an artificial variable to be minimized, indices RMS, XXX, BAS and i indicate, respectively, regions, commodities, base year and activities (e.g., production, processing, imports etc.), and p_weight is a parameter of weights between 1 and 100 assigned to regions and activities. These weights are necessary to achieve plausible calibrated values and their specification is the outcome of a trial and error process, inspecting results from data calibration and retrying. They depend on the results of global database and trends generation. On the right hand-side of the equation v stands for a variable to be estimated and DATA – for base year data already adjusted at the data preparation and balancing stage. Hence with this equation squared sum over regions and commodities of differences between estimated and observed values (and or quantities), these differences being scaled by the observed data times the weight parameters, is minimized. Respectively, calibrated base year data fits the system of the market equations, given certain parameter values, and resembles the observed data as closely as possible. The activities implied under the i index include quantities of production, human consumption, feed, processing, processed to biofuels, import and export, producer, consumer and market prices, difference between market prices and import prices to reduce differences between physical and Armington aggregation, consolidated gap between producer and market prices, processing margin, trade flows and transport costs.
  
-The process of model solving is navigated with C:\...\CAPRI\gams\arm\data_fit.gms file. Its main function is to assure model solving by keeping the market balances closed and price system consistent. Because of the very large number of equations with the exact similar number of variables (36 thsds) that makes the system of equations square, as well as non-linear formulation of some of the equations, it is very likely that infeasibilities will occur during the model solving. To ensure the feasibility as far as possible, code elements such as widening of variable bounds, once they become binding, reducing non-smoothness of the functional forms and introduction of slack variables are introduced. More detailed information on this process can be found in a technical document by Wolfgang Britz and Heinz-Peter Witzke //Infeasibilities in the market model of CAPRI – how they are dealt with// at [[https://www.capri-model.org/docs/infes.pdf]].+The process of model solving is navigated with C:/.../CAPRI/gams/arm/data_fit.gms file. Its main function is to assure model solving by keeping the market balances closed and price system consistent. Because of the very large number of equations with the exact similar number of variables (36 thsds) that makes the system of equations square, as well as non-linear formulation of some of the equations, it is very likely that infeasibilities will occur during the model solving. To ensure the feasibility as far as possible, code elements such as widening of variable bounds, once they become binding, reducing non-smoothness of the functional forms and introduction of slack variables are introduced. More detailed information on this process can be found in a technical document by Wolfgang Britz and Heinz-Peter Witzke //Infeasibilities in the market model of CAPRI – how they are dealt with// at [[https://www.capri-model.org/docs/infes.pdf]].
  
-After solving the MODEL m_calMarketBas, the calibrated data are stored, new producer prices for agricultural outputs are set, sugar beet prices as a function of – sugar market price – sugar export price (pre-reform) or ethanol market price (post-reform) – processing yield (specific to CUR to calibrate to any set of projected beet prices) – levying model for A- and B- sugar (pre-reform) are calculated, share and shift parameters of CES-functions used in the Armington approach to determine import shares as a function of import prices are defined (file C:\...\CAPRI\gams\arm\cal_armington.gms). Furthermore, energy conversion factors for animal products are defined with MODEL m_fitFeedConv (in file C:\...\CAPRI\ gams\arm\feed_conv_decl.gms).+After solving the MODEL m_calMarketBas, the calibrated data are stored, new producer prices for agricultural outputs are set, sugar beet prices as a function of – sugar market price – sugar export price (pre-reform) or ethanol market price (post-reform) – processing yield (specific to CUR to calibrate to any set of projected beet prices) – levying model for A- and B- sugar (pre-reform) are calculated, share and shift parameters of CES-functions used in the Armington approach to determine import shares as a function of import prices are defined (file C:/.../CAPRI/gams/arm/cal_armington.gms). Furthermore, energy conversion factors for animal products are defined with MODEL m_fitFeedConv (in file C:/.../CAPRI/gams/arm/feed_conv_decl.gms). 
 + 
 +//Data balancing for the simulation year// \\
  
-Data balancing for the simulation year \\ 
 Aim of data calibration for the simulation year aims at generating such quantity, price and other market values (see list below) for the simulation year that they fit the system of equations of the market module and variable and parameter lower and upper bounds, as well as remain as close as possible to the values to which they are calibrated (e.g., trends, estimated with growth rates from the base year, Aglink-COSIMO values, GLOBIOM values etc.). Thus process, basically, follows similar approach as for the base year. There are, however, a few differences. The main is that the model used for calibration is MODEL m_calMarketFin. As the model for base year calibration (MODEL m_calMarketBas), it is defined in cal_models.gms file and includes similar equations of the market model with the exception of NSSQ_ equation. The latter equation is replaced by NSSQ1_. Its major difference from NSSQ_ is that DATA parameter includes not values of the base year, but values projected in trend generation step for some of the factors and values shifted to the simulation year based on assumptions or growth rates for the other factors. Thus, it is used for minimizing the differences between estimated and projected (with trend generation step or growth rates) values of the variables in question. Another difference of NSSQ1_ with NSSQ_ is that it includes the differences in intervention stock changes and excludes the differences in consumer prices and gaps between producer and market prices.  Aim of data calibration for the simulation year aims at generating such quantity, price and other market values (see list below) for the simulation year that they fit the system of equations of the market module and variable and parameter lower and upper bounds, as well as remain as close as possible to the values to which they are calibrated (e.g., trends, estimated with growth rates from the base year, Aglink-COSIMO values, GLOBIOM values etc.). Thus process, basically, follows similar approach as for the base year. There are, however, a few differences. The main is that the model used for calibration is MODEL m_calMarketFin. As the model for base year calibration (MODEL m_calMarketBas), it is defined in cal_models.gms file and includes similar equations of the market model with the exception of NSSQ_ equation. The latter equation is replaced by NSSQ1_. Its major difference from NSSQ_ is that DATA parameter includes not values of the base year, but values projected in trend generation step for some of the factors and values shifted to the simulation year based on assumptions or growth rates for the other factors. Thus, it is used for minimizing the differences between estimated and projected (with trend generation step or growth rates) values of the variables in question. Another difference of NSSQ1_ with NSSQ_ is that it includes the differences in intervention stock changes and excludes the differences in consumer prices and gaps between producer and market prices. 
  
-Before MODEL m_calMarketFin is solved, values of DATA parameter for the simulation year are defined. For example, administrative prices for dairy products and cereals and minimum import prices for cereals (in C:\...\CAPRI\gams\arm\prep_pol.gms) and policy data are defined, market prices, quantity variables are shifted with growth rates (C:\...\CAPRI\gams\arm\shift_quantities.gms) and tariffs are defined. Bounds for tariff variables, market prices, milk fat and protein as well as upper and lower limits on quantity variables are assigned as well. At this point, models to calibrate TRQs and entry price equations (MODEL m_fitTrq) and parameters of equations for the intervention stock changes (MODEL m_trimInterv) are solved as well (now for the simulation year, as before it was solved for base year values).+Before MODEL m_calMarketFin is solved, values of DATA parameter for the simulation year are defined. For example, administrative prices for dairy products and cereals and minimum import prices for cereals (in C:/.../CAPRI/gams/arm/prep_pol.gms) and policy data are defined, market prices, quantity variables are shifted with growth rates (C:/.../CAPRI/gams/arm/shift_quantities.gms) and tariffs are defined. Bounds for tariff variables, market prices, milk fat and protein as well as upper and lower limits on quantity variables are assigned as well. At this point, models to calibrate TRQs and entry price equations (MODEL m_fitTrq) and parameters of equations for the intervention stock changes (MODEL m_trimInterv) are solved as well (now for the simulation year, as before it was solved for base year values).
  
-As m_calMarketBas model, m_calMarketFin model is solved by minimizing SSQ value by applying the approach of assuring feasibility via data_fit.gms file. After the solution is found and energy conversion factors for animal products are defined with MODEL m_fitFeedConv, the results are stored in C:\...\CAPRI\ output\results\baseline\data_market_1230.gdx.+As m_calMarketBas model, m_calMarketFin model is solved by minimizing SSQ value by applying the approach of assuring feasibility via data_fit.gms file. After the solution is found and energy conversion factors for animal products are defined with MODEL m_fitFeedConv, the results are stored in C:/.../CAPRI/ output/results/baseline/data_market_1230.gdx.
  
 ====Stage II: Elasticity trimming==== ====Stage II: Elasticity trimming====
Line 626: Line 626:
 At first, parameters for land use market are calculated based on data from FAO world food market model. Among them are land use classes, crop yields, land demand of non-crop activities, areas used for fodder and average land price, total energy use for feeding and producer price of feed. Next, starting elasticity values, as well as their lower and upper bounds are loaded (e.g., demand elasticities used in SPEL/MFSS). Finally, elasticities are trimmed. At first, parameters for land use market are calculated based on data from FAO world food market model. Among them are land use classes, crop yields, land demand of non-crop activities, areas used for fodder and average land price, total energy use for feeding and producer price of feed. Next, starting elasticity values, as well as their lower and upper bounds are loaded (e.g., demand elasticities used in SPEL/MFSS). Finally, elasticities are trimmed.
  
-Elasticities trimming is controlled by C:\...\CAPRI\gams\arm\trim_par.gms file. The elasticity groups are: for calibration of demand and supply systems, feed demand system, oilseeds crush, oil processing and dairy industry. Elasticities of supply system, oilseeds crushing, oil processing and dairy industries, as well as for feed demand, are estimated with MODEL m_trimElas. It is solved by minimising absolute squares between given and calibrated elasticities including land elasticities (FitElas_) subject to the following constraints: marginal effects from price and quantity for current elasticity estimate (Hess_), homogeneity of degree zero for elasticities in prices (HomogN_), Cholesky decomposition of marginal effects to ensure correct curvature (Chol_), Ensure that own price elasticity exceeds yield elasticity * 1.5 (YieldElas_) and elasticities for total energy and protein intake from feeding (ReqsElas_). +Elasticities trimming is controlled by C:/.../CAPRI/gams/arm/trim_par.gms file. The elasticity groups are: for calibration of demand and supply systems, feed demand system, oilseeds crush, oil processing and dairy industry. Elasticities of supply system, oilseeds crushing, oil processing and dairy industries, as well as for feed demand, are estimated with MODEL m_trimElas. It is solved by minimising absolute squares between given and calibrated elasticities including land elasticities (FitElas_) subject to the following constraints: marginal effects from price and quantity for current elasticity estimate (Hess_), homogeneity of degree zero for elasticities in prices (HomogN_), Cholesky decomposition of marginal effects to ensure correct curvature (Chol_), Ensure that own price elasticity exceeds (yield elasticity * 1.5) (YieldElas_) and elasticities for total energy and protein intake from feeding (ReqsElas_). 
  
 Human consumption elasticities are estimated with MODEL m_trimDem by minimizing absolute squares between given and calibrated elasticities (FitElas_). Apart from the objective function the model includes several equations related to the definition of the demand system as Generalized Leontief, homogeniety of degree zero for elasticities in prices, additivity of income elasticities weighted with budget shares and elasticities for total calorie intake.  Human consumption elasticities are estimated with MODEL m_trimDem by minimizing absolute squares between given and calibrated elasticities (FitElas_). Apart from the objective function the model includes several equations related to the definition of the demand system as Generalized Leontief, homogeniety of degree zero for elasticities in prices, additivity of income elasticities weighted with budget shares and elasticities for total calorie intake. 
Line 636: Line 636:
 The fertilizer flows are also calibrated here. The prior parameters for the fertilizer flows are defined based on the //posterior// mode of the base year, by modifying them with land use changes: the fertilization per ha is computed in the base year situation and then multiplied with the areas in the calibration point. The fertilizer flows are calibrated with the same calibration model as used for the base year in the database tasks. The fertilizer flows are also calibrated here. The prior parameters for the fertilizer flows are defined based on the //posterior// mode of the base year, by modifying them with land use changes: the fertilization per ha is computed in the base year situation and then multiplied with the areas in the calibration point. The fertilizer flows are calibrated with the same calibration model as used for the base year in the database tasks.
  
-The file C:\...\CAPRI\gams\capmod\def_fert_and_requirements.gms defines animal nutrient requirements and the nutrient requirements of the crops given trend forecasted yields. In particular, feed input coefficients are defined and calibrated, days in production process of fattening are defined, and manure output is taken into consideration as an input for fertilizer calibration. Fertilizer calibration is basically a merge of trend based forecasts from the ex-post CAPREG results. The fertilizer need is calculated as a function of yield, and adjusted according to the exogenous assumptions. Furthermore, crop nutrient need factors from trends are scaled and logistic function is used to calculate average growth rate of fertilizer use. The calculations must as well comply with the fertilizer equations of the supply model.+The file C:/.../CAPRI/gams/capmod/def_fert_and_requirements.gms defines animal nutrient requirements and the nutrient requirements of the crops given trend forecasted yields. In particular, feed input coefficients are defined and calibrated, days in production process of fattening are defined, and manure output is taken into consideration as an input for fertilizer calibration. Fertilizer calibration is basically a merge of trend based forecasts from the ex-post CAPREG results. The fertilizer need is calculated as a function of yield, and adjusted according to the exogenous assumptions. Furthermore, crop nutrient need factors from trends are scaled and logistic function is used to calculate average growth rate of fertilizer use. The calculations must as well comply with the fertilizer equations of the supply model.
  
-====tage IV: Initialization and test run====+====Stage IV: Initialization and test run====
  
 After the behavioural blocks of the market model are calibrated (one-by-one), the whole model should be also tested for being correctly calibrated. In essence, the test initializes the model with the data against the model was calibrated, and then executes/solves the market model. In theory, a perfectly calibrated model can be solved in one single iteration, without adjustments in the values of the model variables. That is why the iteration limit is technically set to zero (i.e. not allowing for adjustment in the model variables) for the test solve. In practice, a number of infeasibilities might exist due to the accuracy of the numerical solution. But infeasibilities stemming from rounding errors must be small, so the sum of all infeasibilities gives a good indication on the quality of the model calibration.  After the behavioural blocks of the market model are calibrated (one-by-one), the whole model should be also tested for being correctly calibrated. In essence, the test initializes the model with the data against the model was calibrated, and then executes/solves the market model. In theory, a perfectly calibrated model can be solved in one single iteration, without adjustments in the values of the model variables. That is why the iteration limit is technically set to zero (i.e. not allowing for adjustment in the model variables) for the test solve. In practice, a number of infeasibilities might exist due to the accuracy of the numerical solution. But infeasibilities stemming from rounding errors must be small, so the sum of all infeasibilities gives a good indication on the quality of the model calibration. 
  
-At the final stage, some of the starting values and bounds for the market model are set, and agricultural policy data are loaded, adjusted and extended to the simulation year. The policy data include single area payment scheme, set-aside regulations, differentiation between old and new MSs payments, special national envelopes, Nordic schemes, changes in administrative prices, rural development policy and other major CAP post-2014 instruments. Policy files used for the baseline are located in C:\...\CAPRI\ gams\scen\base_scenarios folder. Their loading into the baseline process is controlled by CAP_2014_2020.gms file. With the data mentioned, the outcome of calibration of the CAPRI market module can be tested. In particular, the market model is solved at "trend values" and, thus, the calibration outcome is checked for fitting to the square system of market model equations. This is controlled by C:\...\CAPRI\.gams\arm\prep_market.gms file.+At the final stage, some of the starting values and bounds for the market model are set, and agricultural policy data are loaded, adjusted and extended to the simulation year. The policy data include single area payment scheme, set-aside regulations, differentiation between old and new MSs payments, special national envelopes, Nordic schemes, changes in administrative prices, rural development policy and other major CAP post-2014 instruments. Policy files used for the baseline are located in C:/.../CAPRI/gams/scen/base_scenarios folder. Their loading into the baseline process is controlled by CAP_2014_2020.gms file. With the data mentioned, the outcome of calibration of the CAPRI market module can be tested. In particular, the market model is solved at "trend values" and, thus, the calibration outcome is checked for fitting to the square system of market model equations. This is controlled by C:/.../CAPRI/.gams/arm/prep_market.gms file.
  
 ====Technical remarks==== ====Technical remarks====
Line 654: Line 654:
 ====Introduction==== ====Introduction====
  
-The supply side models of the CAPRI simulation tool are programming models with an objective function. If we want the optimal solution to coincide with the forecast procuced by the projection tools of CAPTRD, we need to ensure that first and second order optimality conditions (marginal revenues equal to marginal costs, all constraints feasible, and the solution is a maximum point) hold in the calibration point for each of the NUTS 2 or farm type models. The consequences regarding the calibration are threefold:+The supply side models of the CAPRI simulation tool are programming models with an objective function. If we want the optimal solution to coincide with the forecast produced by the projection tools of CAPTRD, we need to ensure that first and second order optimality conditions (marginal revenues equal to marginal costs, all constraints feasible, and the solution is a maximum point) hold in the calibration point for each of the NUTS 2 or farm type models. The consequences regarding the calibration are threefold:
  
   - Elements not projected so far but entering the constraints of the supply models (e.g. feed, fertilization) must be defined in such way that constraints are feasible,   - Elements not projected so far but entering the constraints of the supply models (e.g. feed, fertilization) must be defined in such way that constraints are feasible,
Line 662: Line 662:
 ====Calibrating feed and fertilizer restrictions==== ====Calibrating feed and fertilizer restrictions====
  
-The calibration of feed and fertilization restrictions happens in the file //gams\capmod\def_fert_and_requirement.gms.// As explained above, the requirement functions used in the projection tools are linear approximations for the ones used in the simulation tool; additional constraints restrict the feed mix in the supply modules. +The calibration of feed and fertilization restrictions happens in the file //gams/capmod/def_fert_and_requirement.gms.// As explained above, the requirement functions used in the projection tools are linear approximations for the ones used in the simulation tool; additional constraints restrict the feed mix in the supply modules. 
  
 It is hence necessary to find a //feed mix// in the projected point which exhausts the projected production of non-tradable feed and the projected feed mix of marketable bulk feeds (cereals, protein feed, …), fits in the requirement constraints and leads to plausible feed cost. In order to do so, the feed allocation framework used to construct the base year allocation of feedstuff to animals is re-used. The resulting factors are stored in external files and reloaded by counterfactual runs. It is hence necessary to find a //feed mix// in the projected point which exhausts the projected production of non-tradable feed and the projected feed mix of marketable bulk feeds (cereals, protein feed, …), fits in the requirement constraints and leads to plausible feed cost. In order to do so, the feed allocation framework used to construct the base year allocation of feedstuff to animals is re-used. The resulting factors are stored in external files and reloaded by counterfactual runs.
  
-Similar to animal feed balance, the crop nutrient needs must be consistent with available projected nutrients from various sources. To find such a feasible point, the distribution of various fertilizer sources (manure, mineral fertilizers and crop residues) to crops estimated in the database (CAPREG), is shifted with changes in crop areas to make a first best guess (prior) of the allocation to crops in the baseline. This prior is used as the modal value of a probability density function of a Bayesian estimation, similar to the CAPREG procedure described in a previous section of the documentation. Thus, a crop nutrient allocation is sought that is in some sense “as similar” to the base year estimate as possible. The result of the fertilizer calibration for the baseline is stored in a GDX file for each country, found in the directory “results\fert”, from where it is loaded in simulations (by the file //gams\capmod\load_fert_baseline.gms//).+Similar to animal feed balance, the crop nutrient needs must be consistent with available projected nutrients from various sources. To find such a feasible point, the distribution of various fertilizer sources (manure, mineral fertilizers and crop residues) to crops estimated in the database (CAPREG), is shifted with changes in crop areas to make a first best guess (prior) of the allocation to crops in the baseline. This prior is used as the modal value of a probability density function of a Bayesian estimation, similar to the CAPREG procedure described in a previous section of the documentation. Thus, a crop nutrient allocation is sought that is in some sense “as similar” to the base year estimate as possible. The result of the fertilizer calibration for the baseline is stored in a GDX file for each country, found in the directory “results/fert”, from where it is loaded in simulations (by the file //gams/capmod/load_fert_baseline.gms//).
  
 ====Calibrating the marginal cost functions==== ====Calibrating the marginal cost functions====
Line 705: Line 705:
  
 **Figure 12: CAPRI settings: read in all the country specific gdx files from the results directory (capmod); load a specified symbol (dataout); store the data back into a gdx file with the same name but without country suffix** **Figure 12: CAPRI settings: read in all the country specific gdx files from the results directory (capmod); load a specified symbol (dataout); store the data back into a gdx file with the same name but without country suffix**
-{{::figure_12.png?600|}} \\ Source: own illustration+{{::figure_12.png?600|Source: own illustration}}
  
 Then, the GUI can be used in a standard fashion to manually compare the activity levels reported after calibration with those computed in a baseline reproduction run. Then, the GUI can be used in a standard fashion to manually compare the activity levels reported after calibration with those computed in a baseline reproduction run.
  
baseline_generation.1587797127.txt.gz · Last modified: 2022/11/07 10:23 (external edit)
Except where otherwise noted, content on this wiki is licensed under the following license: CC0 1.0 Universal
CC0 1.0 Universal Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki