Prediction

The aim of the Prediction tool is to take advantage of the signalling circuit activities to distinguish between phenotypes. Depending on the study design, we can perform a two group comparison or a correlation with a continuous variable.

HiPathia Prediction uses the signalling value of mechanism-based biomarkers to compute a SVM prediction model with cross-validation. Moreover, previously obtained models could be used to predict the phenotype of new samples.

The tool can be accessed from the main menu bar, by clicking on the Prediction button, see Workflow for further information.

The main page of a the tool is its filling in form. This form includes all the information and parameters that the tool needs to process a study. The form is divided in different panels:

The type panel allows you to choose the kind of prediction analysis you want to perform. You can choose between two options:

• Train new model: A prediction model is generated with your selected data.

• Test existing model: You can choose an already trained model to predict the phenotype of new samples. Existing prediction models can be selected from your project folders stored in HiPathia's server.

In the input data panel, we must introduce the expression data. The expression data has to be:

• Expression matrix provided by ourselves (see how to upload files in Upload your data).

When we select a gene expression file, the number of samples of this matrix will appear under the “file browser” button as shown below.

The design data panel allows you to choose the kind of experiment you want to perform. You can choose between two kinds of experimental design:

• Two group comparison: The comparison is performed between the two groups described in the experimental design file.
  The experimental design file must include two columns: the first one with the names of the samples, the second one with the class to which each sample belongs.

If the experimental design file contains more than two classes then you have to select the appropriate class for each condition.
Note: the condition 2 will be taken as a reference condition.

• Correlation with continuous variable: A correlation is performed between the values of each effector circuit along the samples and the continuous variable introduced.
  The experimental design file must include two columns: the first one with the names of the samples, the second one with the value of the variable for each one of the samples.

Here we must choose the species of our experiment. You can choose among:

• Human (Homo sapiens)
• Mouse (Mus musculus)
• Rat (Rattus norvegicus).

This panel includes further parameters necessary to run an analysis.
Filter circuits: Check to obtain the circuits that best differentiate your phenotype. This option is only available from Prediction tool.

This panel includes the list of all available pathways in HiPathia. We can select the pathways with which the analysis will be performed.
HiPathia retrieves pathway information from KEGG database. KEGG pathway database is a collection of manually drawn pathway maps representing the knowledge on the molecular interaction, reaction and relation networks. By default all available pathways are selected.
Note: At least one pathway has to be selected.

This panel includes some parameters in order to identify and save our study.

• Output folder: If we want to reorganize our studies we can select the folder in which we want save our report. By default the study will be saved in the home in a folder named “Prediction_train_study-N” if it is a training analysis or “Prediction_test_study-N” if it is a prediction test analysis, N is an integer number.
• Study name: We can give a name to our study. This is very useful to later identify it among the other studies listed in the My studies list.
  The default study name is “Prediction_train_study-N” or Prediction_test_study-N“, N is an integer number.
• Description: We can give a description to our study.

Once the form has been filled in, press the Run analysis button to launch the study. Your study will be listed in the My studies panel, and a panel called Browse my studies will appear showing all your studies and their state. the new study will appear with a queued state then running state. If everything goes well, the state will be done after few minutes(depending on the inputs data and the availability of server). All study states are:

• Queued: The information has been processed and the study has been sent and waits to be processed.
• Running: The study is in progress, study can be cancelled using the stop button.

• Done: The study has ended and the results are available to visualize and download.
• Cancelled: The study was canceled before finishing.
• Error: Sometimes a study can stop returning a error message, you can report and contact us in order to help you to fix it.

The report page of the Prediction tool includes different output results. You can download any table or image showed in the results page by clicking on the name right before it. You can also download the pathway and function matrices by clicking on Circuit values.

The results are divided in different panels:

Here you can find the information about the selected study.

• Name: the study name.
• Description: the description of the current study.
• Tool: the name of the used tool (in this case, is Hipathia).
• Date: study's launching date (MM/DD/AAAA, HH:MM:SS AM/PM format)

Here you can visualize the parameters with which the current study was launched.

• Expression file: The name of the expression file that has been used in the current study.
• Design file: The name of the design file that has been used in the current study.
• Comparison: The groups that have been compared, for example; Normal vs Tumor.
• Paired analysis: Have the input data been paired? No or Yes.
• Species: The species of this experiment; Human (Homo sapiens),Mouse (Mus musculus) or Rat (Rattus norvegicus).

You can download the matrix of circuit activity values by clicking on circuit values. This matrix file indicates for each “effector circuit” the level of activation calculated using Hipathia method for each sample.

Here you can visualize the results from the prediction analysis.

• K-fold cross-validation: The number of equal sized subsamples in which the original sample is randomly partitioned.

• SVM-RBF hypermeter performance

• Test model statistics:

• Split Train pr:

• Split Train roc:

• Split Test pr:

• Split Test roc:

You can download the model statistics.

• Selected features: You can download the filtered paths that best differentiate your phenotype. This section is only available when selecting filter paths option.

The prediction tool is based on a machine learning module, this module of the Hipathia web tool can be summarized as follows:

• Expected input and output:
  • Input features: hipathia circuit values.
  • Input response: 1-D binary array with the same number of samples as the input features.
  • Output 1: CV performance metrics
  • Output 2: The selected features with their respective interaction sign, sorted by their elevance.
    • A positive sign indicates that a given feature pushes the prediction towards the positive class.
  • Output 3: Statistics and ROC, PR curves for typical train test split scenario.
  • Output 4: Probability boxplots for the test set.
• Feature selection:
  • we select the features that best discriminate between the response values by means of the LASSO [4] (using the glmnet r package which implements a fast coordinate descent version of the LASSO [5]).
  • We filter the feature space using those circuits selected in the previous step.
• Hyperparameter search (C cost or margin and γ) of a non-linear SVM [6] with a radial-basis kernel:
  • γ: determines the complexity of the svm frontier.
  • cost: is basically the margin around the frontier established by the svm.
  • method: both γ and margin are obtained using a k-fold cross-validation procedure:
    • for each selection of γ and C we train a svm
    • we compute the mean of the misclassification error over all the folds in the test split
    • we select the best pair of hyperparameters (γ, C), i.e. the ones with the lower CV mean error.
  • From now onwards we fix the features selected by the LASSO and the hyperparameters previously found.
  • The SVM training has been carried out using one of the most powerful libraries to train svm-based models LIBSVM [2] by means of the R interface provided by the package e1071 [1].
• Performance evaluation:
  • We perform a k-fold cross-validation with the features and hyperparameters selected above in order to report the generalization capabilities of the method.
    • The report contains a set of commonly used metrics for classification.
  • We perform a train-test split analysis
    • We randomly select 30% of the samples as the test
    • We train a SVM on the train set using the hyperparameters and features previously found.
    • We provide summary statistics as in the case of the k-fold cross-validation.
    • We plot the ROC and Precision-Recall (PR) curves along with the area under the curve.
    • Note that all curve visualizations have been done using the specialized R package PRROC [3]

The experiment consists in classifying a giving sample as Luminal A or Luminal B (molecular subtype). We use TCGA data (processed by Inma), no pathway filtering was done by hand.

Model Analysis

Hyperparameter search:

CV Performance: CV stats

The most relevant features along with their interaction sign:

Split Analysis

Split Performance: Test stats

PR curve over the test:

ROC curve over the test set:

Probability for the test set: