Paper: Momentum Benefits Non-iid Federated Learning Simply and Provably (ICLR-2024)

Reviewer comment (Novelty):
The same algorithm has been proposed by multiple works [1,2]. FedCM in [1] is exactly the same as the proposed method, and [1] also gives theoretical guarantees under general convex or non-convex smooth assumptions, it is not the case as this work claimed: “However, whether momentum can offer theoretical benefits to FL remains underexplored”.

Author response:
Thanks for pointing this out and bringing FedCM [R1] to our attention. It is indeed the same as our FedAvg-M. We will remove the statements like "resulting in the new algorithm FedAvg-M" from our paper. Please note that the main novelty of our paper does not lie in the algorithm development, but in **clarifying the theoretical improvements brought by momentum**. The FedCM paper cannot justify the theoretical benefits by incorporating momentum. 

To resolve the reviewer's concern, we added a detailed comparison with FedCM in the revised paper, see the highlighted paragraph in Page 6. For reviewer's convenience, we repeat the highlighted paragraph as follows:

*FEDAVG-M coincides with the FEDCM algorithm proposed by Xu et al. (2021b). However, our results outperform that of Xu et al. (2021b) in several aspects. First, our convergence only utilizes the standard smoothness of objectives and gradient stochasticity while Xu et al. (2021b) additionally require bounded data heterogeneity and bounded gradients which are rarely valid in practice, suggesting the limitation of their analysis. Second, the convergence established by Xu et al. (2021b) is significantly weaker than ours and cannot even asymptotically approach the ideal rate $O(1/ \sqrt{NKR})$ in non-convex FL, as demonstrated by the results stated in Table 1.*


The rate comparison established in FedCM and ours are as follows:

FedCM: $\left(\frac{L\Delta ({\sigma^2}+NK{G}^2)}{NKR}\right)^{1/2}+\left(\frac{L\Delta ({\sigma}/{\sqrt{K}}+G)}{R}\right)^{2/3}$.

FedAvg-M: $\left(\frac{L\Delta \sigma^2}{NKR}\right)^{1/2}+\frac{L\Delta}{R}$.

It is observed that the rate established by FedCM is much weaker than FedAvg-M.

Reviewer comment (Evaluation):
The problem complexity is studied over the convergence rate under the same fixed hardware budget, such as communication cost or memory consumption. However, this work forgets to mention the extra requirement over the communication budget or the local storage.

Author response:
First, it is generally true that downlink bandwidth is substantially larger than uplink bandwidth in most practical systems. This significant asymmetry between downlink and uplink capacities suggests that heavier downlink loads may be less problematic compared to lighter uplink loads.

Second, our FedAvg-M does not incur additional downlink load compared to FedAvg when more memory is available. For practical implementation of FedAvg-M, the server does not need to communicate both $x^{r+1}$ and $g^{r+1}$ simultaneously to clients. If clients can store a model copy $x^r$ locally, by only broadcasting the latest server model $x^{r+1}$ from the server to all clients, clients are able to recover the momentum direction through $g^{r+1}=(x^{r+1}-x^r)/\gamma$. By employing the equivalent implementation, FedAvg-M does not suffer from a double in downlink communication. 

Moreover, we would like to remark that the doubling in local storage is customary for advanced FL algorithms beyond FedAvg, see, e.g., Scaffold [R2], Mime [R3]. In fact, as long as the local update rule is not the vanilla SGD, the double local storage can be inevitable to store optimizer states, e.g., momentum state.  

To resolve the reviewer's concern, we added comment in Section 3 of the revised paper. The comment is *"Notably, no extra downlink commmunication cost is needed if clients store the last iterate model $x^r$ so that momentum $g^{r+1}$ can recovered through $(x^{r+1}-x^r)/\gamma$."*

Reviewer comment (Evaluation):
The costs of the downlink communication may be two times more expensive than other methods since the proposed method requires the broadcast of the extra averaged gradient information from the center server to each client, together with the new model parameters.

Author response:
First, it is generally true that downlink bandwidth is substantially larger than uplink bandwidth in most practical systems. This significant asymmetry between downlink and uplink capacities suggests that heavier downlink loads may be less problematic compared to lighter uplink loads.

Second, our FedAvg-M does not incur additional downlink load compared to FedAvg when more memory is available. For practical implementation of FedAvg-M, the server does not need to communicate both $x^{r+1}$ and $g^{r+1}$ simultaneously to clients. If clients can store a model copy $x^r$ locally, by only broadcasting the latest server model $x^{r+1}$ from the server to all clients, clients are able to recover the momentum direction through $g^{r+1}=(x^{r+1}-x^r)/\gamma$. By employing the equivalent implementation, FedAvg-M does not suffer from a double in downlink communication. 

Moreover, we would like to remark that the doubling in local storage is customary for advanced FL algorithms beyond FedAvg, see, e.g., Scaffold [R2], Mime [R3]. In fact, as long as the local update rule is not the vanilla SGD, the double local storage can be inevitable to store optimizer states, e.g., momentum state.  

To resolve the reviewer's concern, we added comment in Section 3 of the revised paper. The comment is *"Notably, no extra downlink commmunication cost is needed if clients store the last iterate model $x^r$ so that momentum $g^{r+1}$ can recovered through $(x^{r+1}-x^r)/\gamma$."*

Reviewer comment (Novelty):
Despite the doubt about proof, the method itself is not novel. Embedding local or global momentum seems straightforward. Meanwhile, considering the broad application scenarios of FL, the experiment is too simple, which is not valid to show the effectiveness of the method. However, if the proof can be validated, those shortages can be largely mitigated.

Author response:
We respectively disagree with the reviewer on this point. The main novelty in our paper lies in **proving the theoretical improvements brought by momentum**, not in proposing new momentum variants of FL algorithms. While momentum has been widely used in federated learning empirically, it is still unknown in literature how momentum can benefit federated learning theoretically. Our paper focused on this open question and proved two novel results:

- **Momentum can remove the influence of data heterogeneity**. When all clients participate in FL, momentum enables FedAvg to get rid of the restrictive data heterogeneity assumption without any impractical algorithmic structure. 

- **Momentum can theoretically speed up FL algorithm**. We have shown that FedAvg, Scaffold, and their variance-reduced variants with momentum achieve faster convergence rate than without momentum. In fact, all achieved results are state-of-the-art compared to existing algorithms. 

With the above two contributions, we are the **first**, to our best knowledge, to validate the usage of momentum in federated learning in theory, which we believe is a novel and important contribution to the community.

Reviewer comment (Theory):
I am totally not sure how to build the relationship between ∇fi(x) and ∇f(x) without the bounded data heterogeneity assumption. I.e., without the popular bounded data heterogeneity assumption, how the local gradient information contribute to the global convergence? Can you briefly explain it?

Author response:
`Q1`: How to build the relationship between $\nabla f_i(x)$ and $\nabla f(x)$ without the bounded data heterogeneity assumption?

A: We briefly present the high-level idea of FedAvg-M in the full participation setting. Note that when the local learning rate $\eta\to 0$ and global learning rate $\gamma$ is fixed, FedAvg will reduce to minibatch SGD over all the clients, i.e., SGD over $f(x)$ with batch size $NK$. In this case, the heterogeneity assumption $\zeta$ is known to be unnecessary. Intuitively, in FedAvg-M, our strategy is also to show that the local iterates will not drift too far away from the initialization in each communication round (Lemma 9) if $\eta$ is properly small with the help of momentum.

Reviewer comment (Theory):
I checked the proof of Theorem 1 (Theorem 11 in the Appendix), and the building of the above relation can track the source back to Lemma 5. However, the very second inequality of the proof of Lemma 5 (Page-14), which builds the above relationship, seems to be wrong to me. Considering the updating step is x^{r+1} = x^{r} – γg^{r+1}, the item −γ||∇f(x^{r})||2 comes out from nowhere, and the sign of the item γ⟨∇f(x^{r}),g^{r+1}⟩ should be negative, and it is the real descent item. Then, without Lemma 5, the most important result in this work, Theorem 1, may be wrong.

Author response:
`Q2`: Lemma 5 seems incorrect.

A: As pointed out and confirmed by reviewer ELLH in the part of "Typos", the second line in Lemma 5 is merely a typo and **does not affect the overall validity**. The correction should be
$$
f(x^{r+1})\leq f(x^r)+\langle \nabla f(x^r),x^{r+1}-x^r\rangle +\frac{L}{2}\\|x^{r+1}-x^r\\|^2 = f(x^r)-\gamma  \\|\nabla f(x^r)\\|^2+\gamma  \langle \nabla f(x^r),\nabla f(x^r)-g^{r+1}\rangle+\frac{L\gamma^2}{2}\\|g^{r+1}\\|^2.
$$
We have addressed this in our revision.

Reviewer comment (Theory):
This work uses Young’s inequality four times (Page 14, 17, 22 and 25), but actually, I am not really sure how to get the derived inequality by applying Young’s inequality each time. Please elaborate on each one.

Author response:
`Q3`: How to use Young inequality?

A: In page 14, $\gamma  \langle \nabla f(x^r),\nabla f(x^r)-g^{r+1}\rangle \leq \frac{\gamma }{2}\\|\nabla f(x^r)\\|^2+\frac{\gamma }{2} \\|\nabla f(x^r)-g^{r+1}\\|^2$. 

In page 17 and 22, $\mathbb{E} [\\|\nabla f_i(x^r)\\|^2] \leq (1+q)\mathbb{E}[\\|\nabla f_i(x^{r-1})\\|^2]+(1+q^{-1})\\| \nabla f_i(x^r)-\nabla f_i(x^{r-1})\\|^2 \leq (1+q)\mathbb{E}[\\|\nabla f_i(x^{r-1})\\|^2]+(1+q^{-1})L^2\mathbb{E}[\\|x^r-x^{r-1}\\|^2]$.

In page 25, $\left(1-\frac{S}{N}\right)\frac{1}{N}\sum_{i=1}^N\mathbb{E}[\\|c_i^r-\nabla f_i(x^r)\\|^2]\leq \left(1-\frac{S}{N}\right)\frac{1}{N}\sum_{i=1}^N\mathbb{E}\left[\left(1+\frac{S}{2N}\right)\\|c_i^r-\nabla f_i(x^{r-1})\\|^2+\left(1+\frac{2N}{S}\right)\\| \nabla f_i(x^r)-\nabla f_i(x^{r-1})\\|^2\right]$ and $\\| \nabla f_i(x^r)-\nabla f_i(x^{r-1})\\|^2 \leq L^2\\|x^r-x^{r-1}\\|^2$.

Reviewer comment (Theory):
Basically, each time when you use the Young’s inequality, you will use the relation, E||x^{r} – x{r−1}||^2 <= γ^{2}E(ξ_{r-1} + E||∇f(x^{r-1})||^{2}) in next step, for example, it has been used to derive the second inequality in Page 22. It is not true to me without any assumptions. Basically, you are saying the averaged gradient can be bounded by the true gradient with some error, this is basically a (new) variant assumption of the assumption you claim you have abandoned (bounded data heterogeneity assumption). I believe you should not abandon the more standard bounded data heterogeneity assumption.

Author response:
`Q4`: About the second inequality in page 22 $\mathbb{E}||x^{r} – x^{r−1}||^2 \leq 2\gamma^{2}\mathbb{E}(\mathcal{E}_{r-1} + \mathbb{E}||\nabla f(x^{r-1})||^{2})$.

A: First, we would like to emphasize that **we did not use any additional assumption**. Note that by the definition of $\mathcal{E}_{r-1}$ in the beginning of Appendix A, we have 

$\mathbb{E}\\|x^r-x^{r-1}\\|^2 = \gamma^2\mathbb{E}\\|g^r\\|^2 \leq 2\gamma^2 \mathbb{E}(\\|g^r-\nabla f(x^{r-1})\\|^2 + \\|\nabla f(x^{r-1})\\|^2) = 2\gamma^2 (\mathcal{E}_{r-1}+\mathbb{E}[\\|\nabla f(x^{r-1})\\|^2])$.

And we derive a recursive bound for $\mathcal{E}_{r-1}$ in Lemma 8. Therefore, we do not need the bounded heterogeneity assumption.

Reviewer comment (Theory):
I checked the mentioned work VRL-SGD that can handle unbounded data heterogeneity. It is not published, and the soundness of the proof is being doubted in the previously peer-reviewing stage. You should be careful to present the conclusions of VRL-SGD as formal results in your work.

Author response:
`Q5`: About VRL-SGD.

A: Thanks for pointing this out and we have made additional comments on the work in our revision, as highlighted in the red color in Table 1. For reviewer's convenience, we repeat the comments as follows: 
*The works have not been published in peer-reviewed venues.*

Reviewer comment (Theory):
The final convergence rate achieved by the authors does not sufficiently account for the impact of the momentum coefficient. Please clarify this issue.

Author response:
**1. Influence of momentum coefficient**

The momentum coefficient does not appear in the final convergence rate because we have **set up a concrete and proper value for $\beta$** when deriving the convergence rate. For example, when establishing the convergence of FedAvg-M in Theorem 11 in the appendix, we first prove that

$$\frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^r)\|^2 = O\left(\frac{L\Delta}{R} + \frac{L\Delta}{\beta R} + \frac{\beta \sigma^2}{NK} + (\beta \eta K L)^2 G_0\right)$$

which reflects how $\beta$ would influence the convergence rate. When we set $\beta = O(\sqrt{\frac{NKL\Delta}{\sigma^2 R}})$ and let $\eta \beta L$ be sufficiently small (see Theorem 11 for the value of $\eta \beta L$), we can simplify the convergence rate as

$$\frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}\|\nabla f(x^r)\|^2 = O\left(\sqrt{\frac{L \Delta \sigma^2}{NKR}} + \frac{L\Delta}{R}\right)$$

which is the SOTA convergence rate. Notably, we remark the above annealing $\beta$ is mainly set for theory purpose. 

In practice, we found a proper constant value (e.g., $\beta=0.5, 0.2$) is also capable to attain substantial improvement, as demonstrated by our experiments.

Reviewer comment (Theory):
In fact, FedDyn [1] demonstrates a faster convergence rate compared to the authors' findings in this paper, which is also without the need of clients’ variance assumptions. This observation may highlight the potential limitations in the author's theoretical contributions.

Author response:
**2. FedDyn and our algorithms are not comparable**

We respectively disagree with the comments that FedDyn demonstrates faster convergence rate than our proposed algorithms. **FedDyn and our algorithm belong to different algorithm families, and they are using different oracles in local updates.** Our algorithm use the standard stochastic gradient oracle in local updates, while FedDyn relies on the black-box oracle that gives the exact solutions to local optimization problems (see line 5 in Algorithm 1 of FedDyn). Note that the local optimization problem is generally intractable (particularly for non-convex functions) and thus the oracle required by FedDyn is usually impractical, which outweighs its benefits in convergence if any. If the local optimization problem is not solved exactly, it is not known whether the current rate in FedDyn still hold. For this reason, we believe FedDyn and our algorithms are not comparable. The convergence rate of FedDyn does not hurt our theoretical contribution.

Reviewer comment (Experiments):
The authors' work seriously lacks comparative experiments, including comparison with various momentum-based federated algorithms [2].

Author response:
**3. Empirical comparison with federated adaptive algorithms**

We thank the reviewer for bringing up the federated adaptive algorithms. However, the main contribution in our paper is to clarify the theoretical improvements brought by introducing momentum to FL algorithms. This goal, to our knowledge, has not been achieved by [R1] or any other literature.

Again, we are not aiming to develop adaptive algorithms that can outperform any federated adaptive algorithms in [R1] empirically. All of our experiments are conducted to validate our theoretical findings, i.e., momentum can help remove the influence of data heterogeneity and enhance the convergence rate. Whether our algorithms can outperform [R1] or not does not provide direct evidence to our theoretical findings. We hope the reviewer can understand that the comparison between our algorithms and adaptive algorithms in [R1] is not necessary and beyond the scope of our paper.

Reviewer comment (Evaluation):
This paper lacks a comprehensive discussion regarding the limitations of the proposed algorithms. It is evident, for instance, that SCAFFOLD exhibits a suboptimal performance at very low sampling rates [2], leaving uncertainty regarding the extent to which the authors' improved algorithm can address this issue.

Author response:
**5. Scaffold with low sampling rate**

We thank the reviewer for the sharp observation. When the number of sampled clients $S$ is far less than the number of all clients $N$, i.e., $S \ll N$, all federated learning algorithms listed in Table 2 will suffer from slower convergence, see their rates in Table 2. It implies that slow convergence in the scenario with a low client-sampling ratio is a common issue in federated learning. 

However, incorporating momentum does bring some benefits. For example, incorporating momentum has improved Scaffold's convergence rate from $O(\sqrt{\frac{L \Delta \sigma^2}{S K R}} + \frac{L \Delta}{R}(N/S)^{2/3})$ to $O(\sqrt{\frac{L \Delta \sigma^2}{S K R}} + \frac{L \Delta}{R}(N^{2/3}/S))$, which quantitively illustrates that momentum indeed accelerates the rate. 

We hope the above clarification can relieve the reviewer's concerns. We believe our paper makes a novel and important contribution to the community on understanding the theoretical benefits of momentum in federated learning, which has never been clarified before.

Reviewer comment (Theory):
Post-rebuttal, while many concerns have been addressed (including both the theoretical and experimental analyses of the hyperparameter $\beta$), there remains some concern regarding the authors' claims on their theoretical contributions of the convergence, which is particularly defeated by the faster convergence rate achieved by FedDyn (though FedDyn requires a clients’ solution optimizer).

Author response:
**2. FedDyn and our algorithms are not comparable**

We respectively disagree with the comments that FedDyn demonstrates faster convergence rate than our proposed algorithms. **FedDyn and our algorithm belong to different algorithm families, and they are using different oracles in local updates.** Our algorithm use the standard stochastic gradient oracle in local updates, while FedDyn relies on the black-box oracle that gives the exact solutions to local optimization problems (see line 5 in Algorithm 1 of FedDyn). Note that the local optimization problem is generally intractable (particularly for non-convex functions) and thus the oracle required by FedDyn is usually impractical, which outweighs its benefits in convergence if any. If the local optimization problem is not solved exactly, it is not known whether the current rate in FedDyn still hold. For this reason, we believe FedDyn and our algorithms are not comparable. The convergence rate of FedDyn does not hurt our theoretical contribution.

Reviewer comment (Novelty):
**Algorithm not new**: Contrary to what is claimed in section 3.1 (*"resulting in the new algorithm FEDAVG-M"*), the added momentum is not new: FedCM [[1]](https://arxiv.org/pdf/2106.10874.pdf) is exactly the same algorithm as FedAVG-M, although their theoretical analysis does use the bounded heterogeneity assumptions. Comparison with FedCM rates is lacking in Table 1.

Author response:
**1. Comparison with FedCM**

Thanks for bringing FedCM to our attention. It is indeed the same as our FedAvg-M. We will remove the statements like "resulting in the new algorithm FedAvg-M" from our paper. Please note that the main novelty of our paper does not lie in the algorithm development, but in **clarifying the theoretical improvements brought by momentum**. The FedCM paper cannot justify the theoretical benefits by incorporating momentum. 

To resolve the reviewer's concern, we added a detailed comparison with FedCM in the revised paper, see the highlighted paragraph in Page 6. For reviewer's convenience, we repeat the highlighted paragraph as follows:

*FEDAVG-M coincides with the FEDCM algorithm proposed by Xu et al. (2021b). However, our results outperform that of Xu et al. (2021b) in several aspects. First, our convergence only utilizes the standard smoothness of objectives and gradient stochasticity while Xu et al. (2021b) additionally require bounded data heterogeneity and bounded gradients which are rarely valid in practice, suggesting the limitation of their analysis. Second, the convergence established by Xu et al. (2021b) is significantly weaker than ours and cannot even asymptotically approach the ideal rate $O(1/ \sqrt{NKR})$ in non-convex FL, as demonstrated by the results stated in Table 1.*


The rate comparison established in FedCM and ours are as follows:

FedCM: $\left(\frac{L\Delta ({\sigma^2}+NK{G}^2)}{NKR}\right)^{1/2}+\left(\frac{L\Delta ({\sigma}/{\sqrt{K}}+G)}{R}\right)^{2/3}$.

FedAvg-M: $\left(\frac{L\Delta \sigma^2}{NKR}\right)^{1/2}+\frac{L\Delta}{R}$.

It is observed that the rate established by FedCM is much weaker than FedAvg-M.

Reviewer comment (Theory):
**Surprising rates for the VR variants**: [[2]](https://link.springer.com/article/10.1007/s10107-022-01822-7) states that *"every randomized algorithm requires $\mathcal O \left( \frac{\Delta L \sigma}{\epsilon^3} + \frac{\Delta L}{\epsilon^2} + \frac{\sigma^2}{\epsilon^2} \right)$  oracle queries"*, however, the rate reported in Table 1 and Theorem 2 for FedAVG-M-VR seems to improve on this lower bound as it leads to an oracle complexity of $\mathcal O \left( \frac{\Delta L \sigma}{NK \epsilon^3} + \frac{\Delta L}{\epsilon^2} \right)$. Setting aside the variance-reduction effect of running a distributed algorithm (leading to the $NK$ term), is it normal to get rid of the $\frac{\sigma^2}{\epsilon^2}$ term or am I wrongly worried? (This question also holds for the VR version of SCAFFOLD-M)

Author response:
**2. VR rates**

Thanks for the sharp observation. **There is no mismatch between our rate and that in [R1]. The difference lies in the number of data batches used in the initialization stage** (i.e., the the number of data batches in the very first iteration). In the paper we set initial number of batches $B$ as $\Theta\left((NKR)^{1/3}(\frac{\sigma^2}{L\Delta})^{4/3}\right)$ for simplicity so we can get rid of the additional $\sigma^2/\epsilon^2$ term. On the other hand, if we set initial batch size as $\Theta(NKR)$ as constrained by the setup in [R1], our rate will match the lower bound stated in [R1]. 

To resolve the reviewer's concern, we also considered the setting with initial batch size $\Theta(NKR)$ and derived the convergence rate, see Theorem 15 and Theorem 22 in the revised paper. We also list the new convergence rate in Table 1. For reviewer's convenience, we put the comparison in FedAvg-M-VR as follows:


For initial batchsize $B=\Theta\left((NKR)^{1/3}(\frac{\sigma^2}{L\Delta})^{4/3}\right)$, the rate is $\left(\frac{L\Delta \sigma}{NKR}\right)^{2/3}+\frac{L\Delta}{R}$.
For initial batchsize $B=\Theta(NKR)$, the rate is $\left(\frac{L\Delta \sigma}{NKR}\right)^{2/3}+ \frac{\sigma^2}{NKR}+\frac{L\Delta}{R}$.

Reviewer comment (Evaluation):
**Additional downlink load**: Although no additional client-to-server communication is necessary, the server-to-client communications are doubled in size with the addition of the momentum for FedAVG-M.

Author response:
**4. Downlink load**

First, it is generally true that downlink bandwidth is substantially larger than uplink bandwidth in most practical systems. This significant asymmetry between downlink and uplink capacities suggests that heavier downlink loads may be less problematic compared to lighter uplink loads.

Second, our FedAvg-M does not incur additional downlink load compared to FedAvg when more memory is available. For practical implementation of FedAvg-M, the server does not need to communicate both $x^{r+1}$ and $g^{r+1}$ simultaneously to clients. If clients can store a model copy $x^r$ locally, by only broadcasting the latest server model $x^{r+1}$ from the server to all clients, clients are able to recover the momentum direction through $g^{r+1}=(x^{r+1}-x^r)/\gamma$. By employing the equivalent implementation, FedAvg-M does not suffer from a double in downlink communication.

To resolve the reviewer's concern, we added comment in Section 3 of the revised paper. The comment is *"Notably, no extra downlink commmunication cost is needed if clients store the last iterate model $x^r$ so that momentum $g^{r+1}$ can recovered through $(x^{r+1}-x^r)/\gamma$."*

Reviewer comment (Theory):
**Lacking discussion on link between $R$ and $\beta$**: While a constant step-size can be considered for sufficiently high values of $R$, the direct corollary is that, before that regime arrives, Theorem 1 sets a value of $\beta=1$, meaning that the theory seems to predict that the momentum could only be used for sufficiently long training. However, experiments in Fig. 2 seem to show that using a momentum would help even if the training stopped early.

Author response:
**5. Link between $R$ and $\beta$**

In Theorem 1 we set $\beta=1$ when $R$ is small for simplicity. However, we remark that $\beta$, in principle, can be any $\Theta(1)$ constant that is less than $1$ instead of being exactly $1$. The convergence rates will not affected by this choice, which can be easily derived with very minor modifications to the proof.

To resolve the reviewer's concern, we have revised the choise of $\beta$ in Theorems 1, 11, 15, 19 and 22.