version 2.2

EVERY ORDINARY SYMPLECTIC ISOGENY CLASS IN

POSITIVE CHARACTERISTIC IS DENSE IN THE MODULI

Ching-Li Chai

Abstract. We prove that any ordinary symplectic separable isogeny class in the moduli space of principally polarized abelian varieties over a field of positive characteristic is dense in the Zariski topology.

Contents

Introduction 1. Strategy and methods of the proof

2. Calculation at the 0-dimensional cusp

3. Reduction to the Hilbert-Blumenthal case

4. Calculation at smooth ordinary points over finite fields

5. Inspection at the supersingular points

Introduction

Let k be an algebraically closed field of characteristic p > 0. Let g ≥ 1 be a positive integer and let A_{g} be the moduli stack of principally polarized abelian varieties of dimension g. Suppose that x ∈ A_{g}(k) corresponds to an abelian variety A_{x}. Let D be a positive integer not divisible by p. Consider the set G^{(p)}(x) ⊆ A_{g}(k) (resp. G_{D}(x) ⊂ A_{g}(k) ) consisting of all points y ∈ A_{g}(k) such that there exists an isogeny ϕ : A_{y} → A_{x} with ϕ^{∗}(pol_{A_{x}}) = m · pol_{A_{y}} for some positive integer m, (m, p) = 1 (resp. m | D^{N} for some non-negative integer N). We shall refer to G^{(p)}(x) (resp. G_{D}(x) ) as the prime-to-p (resp. D-power) Hecke orbit of x in A_{g}. This paper was started by the following question: When is the countable subset G^{(p)}(x) (resp. G_{D}(x) ) Zariski dense in A_{g}(k)? Clearly a necessary condition for either question is that x has to be ordinary.

Otherwise G^{(p)}(x) and G_{ℓ}(x) will be contained in the zero locus of the determinant of the Hasse-Witt matrix. The question is whether this condition is also sufficient. Lacking any counterexample, an optimist would be inclined to form the opinion after some cautious thought that the answer should be ‘yes’. We formulate this expectation as

Typeset by AMS-TEX

This research is partially supported by grant DMS90-02574 from the National Science Foundation and by a grant from the National Science Council of Taiwan, R.O.C.

Question (Q 1). Given an ordinary principally polarized abelian variety A_{0} over an algebraically closed field k of characteristic p > 0 , denote by G^{(p)}(A_{0}) the set of all principally polarized abelian varieties A′ over k such that there exists an isogeny β : A′ → A_{0} which preserves the principal polarizations up to a positive integer prime to p. Is the set G^{(p)}(A_{0}) Zariski dense in the moduli stack A_{g}/k classifying principally polarized abelian varieties of dimension g over k?

Notice that in (Q 1) we can substitute “isogeny” by “prime-to-p quasi-isogeny” without changing the set G^{(p)}(A_{0}). By prime-to-p quasi-isogenies we mean the groupoid generated by prime-to-p isogenies. There is a several-prime version of (Q 1):

Question (Q 1)_{D}. Given an ordinary principally polarized abelian variety A_{0} over an algebraically closed field k of characteristic p > 0. Let D be a positive integer not divisible by p. Denote by G_{D}(A_{0}) the set of all principally polarized abelian varieties A′ over k such that there exists an isogeny β : A_{0} → A′ preserving the principal polarizations up to a divisor of a power of D. Is the set G_{D}(A_{0}) Zariski dense in the moduli stack A_{g}/k classifying principally polarized abelian varieties of dimension g over k?

Notice that we can replace “isogeny” by “D-power quasi-isogeny in (Q 1)_{D}”. Here “D-power quasi-isogenies” is the groupoid generated by isogenies whose kernels are killed by some power of D.

The main result of this article confirms the optimists’ prediction about (Q 1). Namely for any ordinary A_{0} , the countable subset G^{(p)}(A_{0}) is Zariski dense in A_{g}. We actually prove a strong form of (Q 1)_{D}: for any prime number ℓ ≠ p the ℓ-power Hecke orbit of any ordinary point in A_{g}(k) is Zariski dense in A_{g}. The same is also true for general polarization types. See Theorem 2 at the end of this paper. Of course one could have asked the same questions in characteristic zero. But then the answers are readily available via complex analytic uniformization: for any point τ in the period domain, Sp_{2g}(ℤ[1/ℓ]) · τ is already dense in period domain with respect to the metric topology by strong approximation.

One can also consider the question on whether a prime-to-p Hecke orbit in characteristic p is dense in the more general context of Shimura varieties. Suppose that G is a connected reductive linear algebraic group over ℚ, X is a G(ℝ)-conjugacy class of ℝ-homomorphisms h : 𝕊 → G. Here 𝕊 = Res_{ℂ/ℝ}𝔾_{m} is the Weil restriction of scalars of 𝔾_{m} from ℂ to ℝ, and the pair (G, X) satisfies Deligne’s axioms in [De, 2.1.1]. Let Sh(G, X) be the canonical model of the Shimura variety associated to (G, X), defined over the Shimura field E = E(G, X). If 𝔭 is a finite place of E(G, X) lying over a prime number p, and suppose that the group G(ℚ_{p}) has a hyperspecial maximal compact subgroup K_{p}. Under these conditions Langlands and Rapoport [LR] conjectured that Sh(G, X)/K_{p} has good reduction at 𝔭, and Milne [Mi1, Mi2] has given a conjectural description of its canonical integral model. For the sake of discussion, let us assume that the canonical integral model exists. Then G(𝔸_{f}^{(p)}) = ∏′_{ℓ≠p} G(ℚ_{ℓ}) operates on the mod 𝔭 fiber (Sh(G, X)/K_{p})⊗_{O_{E}} κ(𝔭) of the canonical integral model of Sh(G, X)/K_{p} via algebraic correspondences. It is expected that (Sh(G, X)/K_{p}) ⊗_{O_{E}} κ(𝔭) has a natural stratification, which is preserved by the G(ℚ_{ℓ})-action. Unless a given stratum has a natural fibration structure compatible with the Hecke action coming from G(𝔸_{f}^{(p)}), one expects that the Zariski closure of any point in a given stratum contains an irreducible component of the whole stratum. Especially, if (Sh(G, X)/K_{p}) ⊗_{O_{E}} κ(𝔭) has ordinary points, one expects that the Zariski closure of the G(𝔸_{f}^{(p)})-orbit of an ordinary point is actually equal to (Sh(G, X)/K_{p}) ⊗_{O_{E}} κ(𝔭) . Here we have to clarify the meaning of “ordinary points”. Conjecturally, over (Sh(G, X)/K_{p})⊗_{O_{E}} κ(𝔭) there is a natural family of F -crystals E_{ρ} attached to any ℚ-rational representation ρ of G. We say that a point x of (Sh(G, X)/K_{p}) ⊗_{O_{E}} κ(𝔭) is ordinary if the Newton polygon of the F -crystal E_{ρ,x} coincides with its Hodge polygon for every ℚ-rational representation ρ. This more general question is far from being solved. The method we used applies to other Shimura varieties of PEL-type as well. We intend to explore this in another paper.

There is a local version of question (Q 1):

Question (Q 2). Let (A_{1}, λ_{1}) be a principally polarized supersingular abelian variety of dimension g over an algebraically closed field k of characteristic p > 0. To say that A_{1} is supersingular means that A_{1} is isogenous to the product of g copies of a supersingular elliptic curve E_{s} over k, or equivalently that all slopes of the Newton polygon of A_{1} are equal to 1/2. Let 𝔖_{pol,1} be the equicharacteristic deformation space of (A_{1}, λ_{1}) , which by the Serre-Tate theorem is the same as the equicharacteristic deformation space of the formal completion (Â_{1}, λ̂_{1}) of (A_{1}, λ_{1}) along the zero section. The compact p-adic group Aut(Â_{1}, λ̂_{1}) of automorphisms of (Â_{1}, λ̂_{1}) operates naturally on 𝔖_{pol,1}. If ℨ ⊆ 𝔖_{pol,1} is a closed formal subscheme of 𝔖_{pol,1} stable under an open subgroup of (Â_{1}, λ̂_{1}) such that the Hasse invariant (= determinant of the Hasse-Witt matrix of the universal abelian scheme over 𝔖_{pol,1} ) is not a zero-divisor on ℨ, is ℨ necessarily equal to 𝔖_{pol,1} itself?

We first approached the global question (Q 1) by reducing it to (Q 2), a question in deformation theory. Unfortunately (Q 2) has resisted all my attempts to unravel it. The mysterious action of Aut(Â_{1}, λ̂_{1}) on 𝔖_{pol,1} remains to be understood. At this moment I can only prove a much weakened version of a special case of (Q 2) by a revoltingly complicated calculation. Therefore it seems preferable to wait until better theorems are available, rather than publishing this hopelessly incomplete result prematurely. However it may be worthwhile to formulate this local question in a more general setting. For any p-divisible formal group G_{0} over a field of characteristic p, there is a natural action of the group of automorphisms of G_{0} on the deformation space of G_{0}. As far as I know, the paper [LT] by Lubin and Tate is the first to discuss group actions of this sort. The action on the Lubin-Tate moduli space on the generic fiber was closely studied in the recent paper [GH]. They also appeared in the important recent work of Rapoport and Zink. It seems that this action is quite fundamental and deserves further investigation. Even in the case when G_{0} is a one-dimensional formal group of finite height, this action is not at all well-understood. For instance the equivariant cohomology groups of the group of units of the deformation space contain deep information about the stable homotopy group of spheres, because they are closely related to the chromatic spectral sequence. The analogue of question (Q 2) is also unresolved in this case, see the end of [Ch 3] for a precise formulation of the question. Hopefully the question (Q 2) will be eventually solved as our knowledge about these actions accumulates.

I would like to express my gratitude to J. Milne, whose conjecture on the existence of canonical integral models of Shimura varieties motivated me to ask (Q 1); to G. Faltings for the idea of reduction to supersingular points ; to M. Larsen for many hours of stimulating and enjoyable discussion on (Q 1), and also for relating Faltings’ idea to me; to F. Oort for patiently going through the proof and pointing out mistakes, and also for stimulating comments and encouragement; to B. Gross and M. Hopkins, for their inspiring paper [GH] and also for discussions and email communications about the p-adic period map; to P. Deligne for pointing out a slip in an earlier version; to A. Neeman and W. Messing for useful conversations and comments; and to the referee for a very careful reading of the manuscript and many useful suggestions. It is a great pleasure to acknowledge my intellectual debt to all of them. A part of the work on this paper was done while the author was on sabbatical leave from the University of Pennsylvania during the year 1992/93. I would like to thank Academia Sinica and the Mathematical Sciences Research Institute for their hospitality and for providing excellent working environment.

This paper is organized as follows. The strategy of the proof of the main results Theorems 1 and 2 is described in §1. There are three kinds of local calculations involved: at the boundary of A_{g}, at ordinary points over finite fields and at supersingular points. These are carried out in §2, §4 and §5 respectively. Another ingredient is a trick of reduction to the Hilbert-Blumenthal moduli space. This is explained in §3. Readers in a hurry can read §1 and the statements of propositions and theorems to get the idea of the proof.

1. Strategy and methods of the proof

Let k be an algebraically closed field of characteristic p > 0. Let g ≥ 1 be a positive integer. Let A_{g} = A_{g}/k be the moduli stack classifying principally polarized abelian varieties of dimension g over Spec(k) . For a point x ∈ A_{g}(k) corresponding to a principally polarized abelian variety (A_{x}, λ_{x}) , the expectation (Q 1) in the introduction states that

(Q 1). The countable set

G^{(p)}(x) = \{ y ∈ A_{g}(k) | \begin{matrix}∃ an isogeny ϕ : A_{y} → A_{x} and m ∈ ℕ, \\ s. t. (m, p) = 1 and ϕ^{∗}(λ_{x}) = m λ_{y} \end{matrix}\}

is Zariski dense in A_{g} if A_{x} is ordinary.

One problem in dealing with problems like (Q 1) is that the usual machinery of algebraic geometry is not designed to deal with problems about finding the Zariski closure of a countable set of points. At a first look, it is not even clear whether there exists any ordinary point x ∈ A_{g}(k) such that G^{(p)}(x) is dense if g ≥ 2 . When g = 1 , it is easy to see that for any prime number ℓ ≠ p, G_{ℓ}(x) is infinite if and only if A_{x} is an ordinary elliptic curve, say by using the theory of canonical liftings. (Later on in this section we shall see that for any g ≥ 1 , G^{(p)}(x) is finite if and only if x is supersingular.) Thus (Q 1)_{ℓ} holds for every ℓ ≠ p in this case for the reason of dimension.

We first give an example, due to Michael Larsen, of a point x ∈ A_{g}(k) such that G_{ℓ}(x) is Zariski dense in A_{g}.

Example. (M. Larsen) Assume that k = \overline{𝔽_{p}} and let E be an ordinary elliptic curve over k, with its natural principal polarization λ_{E}. Let (A, λ) = (E, λ_{E})^{⊕g}, g ≥ 2 , and let x ∈ A_{g}(k) be the corresponding point. Then G_{ℓ}(x) is Zariski dense in A_{g} for any prime number ℓ ≠ p.

Proof of Example. First of all, we show that x is a smooth point of the Zariski closure Z_{ℓ} of G_{ℓ}(x) . The ℓ-power Hecke operates on A_{g} via algebraic correspondences. Although this is not exactly a group action, it comes from the group GSp_{2g}(ℚ_{ℓ}) , and G_{ℓ}(x) is very much like an orbit of x under GSp_{2g}(ℚ_{ℓ}) . The usual argument which shows that any orbit for a connected algebraic group acting on an algebraic variety is smooth applies in this case: The smooth locus of Z_{ℓ} is a nonempty open subscheme Z_{ℓ,sm} of Z_{ℓ}, which is stable under ℓ-power Hecke correspondences and contains points of G_{ℓ}(x) by definition. Hence x ∈ Z_{ℓ,sm}.

Let O be the endomorphism ring of E. It is well-known that O is an order in an imaginary quadratic number field, and O ⊗_{ℤ} ℤ_{p} ≅ ℤ_{p} × ℤ_{p}. The semigroup

S = GL_{g}(O[1/ℓ]) ∩ M_{g×g}(O) ∩ GSp_{2g}(ℚ)

operates on (A, λ) = (E, λ_{E})^{⊕g} as endomorphisms, which are ℓ-power isogenies and preserve the polarization up to ℓ-power multiples. Also the elements in the semigroup S induce ℓ-power Hecke correspondences on A_{g}/k. Consequently the formal completion Z_{ℓ,x}^{∧} ⊆ A_{g,x}^{∧} of Z_{ℓ} at x is stable under the natural action of S on the formal completion A_{g,x}^{∧} of A_{g}/k at x. Since O ⊗_{ℤ} ℤ_{p} ≅ ℤ_{p} × ℤ_{p}, GL_{g}(O ⊗_{ℤ} ℤ_{p})∩GSp_{2g}(ℚ_{p}) is isomorphic to GL_{g}(ℤ_{p}), and S is dense in GL_{g}(ℤ_{p}) under this isomorphism. Therefore the formal subscheme Z_{ℓ,x}^{∧} of A_{g,x}^{∧} is stable under the action of GL_{g}(ℤ_{p}) .

We will only use a tiny part of the above information at the level of the tangent space, namely that the tangent space T_{x}Z_{ℓ} of Z_{ℓ} at x is stable under the natural action of GL_{g}(ℤ_{p}) on T_{x}(A_{g}/k) . But we know that the action of GL_{g}(ℤ_{p}) on T_{x}(A_{g}/k) factorizes through the natural surjection GL(ℤ_{p}) ↠ GL_{g}(𝔽_{p}) . Moreover as a representation of GL_{g}(𝔽_{p}) the tangent space T_{x}(A_{g}/k) is isomorphic to the subspace of symmetric elements in k^{⊕g} ⊗_{k} k^{⊕g}. Now it is well known that the second symmetric product of the standard representation of GL_{g}(𝔽_{p}) is absolutely irreducible if p > 2. (This can be verified either directly, or using general results on representations of Chevalley groups due to Curtis. See for instance [Bo], especially Theorem 6.4 and Corollary 7.3.) If p > 2, the invariant subspace T_{x}Z_{ℓ} being nonzero, it has to be equal to T_{x}(A_{g}/k), and we conclude that Z_{ℓ} = A_{g}/k when p > 2.

When p = 2 , one can still conclude that Z_{ℓ} = A_{g}/k. Since the purpose of this example is to illustrate a general principle to be explained next, we shall only indicate how a proof can be supplied. There are at least two ways to do this. The first way is to use the stronger information that Z_{ℓ,x} is stable under the action of GL_{g}(ℤ_{p}) , and use Serre-Tate coordinates on A_{g,x} to perform computation with higher order deformations. This method of using Serre-Tate coordinates will be explored in §4. Another way is to observe that Z_{ℓ} contains the diagonally embedded modular curve, therefore the Zariski closure Z_{ℓ}^{∗} of Z_{ℓ} in the minimal compactification A_{g}^{∗}/k of A_{g}/k contains the 0-dimensional cusp “\sqrt{−1}∞ · I_{g}”. The result of §2 then implies that Z_{ℓ} = A_{g}/k. □

A close examination of the proof of this example reveals a general method of getting information about the Zariski closure Z of G^{(p)}(x) and about the Zariski closure Z_{ℓ} of G_{ℓ}(x) . This is also the only method we know of. Let Z^{∗} be the Zariski closure of Z in the minimal compactification A_{g}^{∗} of A, or equivalently the Zariski closure of G^{(p)}(x) in A_{g}^{∗}. Clearly if a point y ∈ Z^{∗}(k) ⊆ A_{g}^{∗}(k) is stable under an algebraic correspondence γ coming from GSp_{2g}(𝔸_{f}^{(p)}) , then the formal completion Z_{y}^{∗∧} ⊆ A_{g,y}^{∗} of Z^{∗} at y is stable under γ as well. Hence if a point y ∈ Z^{∗∧}(k) has a large “stabilizer subgroup” in GSp_{2g}(𝔸_{f}^{(p)}) , we get information by analyzing the action of the stabilizer subgroup on A_{g,y}^{∗∧}. When y ∈ A_{g}(k) , this is a problem in deformation theory about arbitrarily high order deformations. When y lies in the boundary of A_{g}, we shall need the theory of degeneration as explained in [FC].

For a ‘general’ point y ∈ A_{g}(k) , we cannot expect its stabilizer subgroup in the the group GSp_{2g}(𝔸_{f}^{(p)}) (resp. GSp_{2g}(ℚ_{ℓ})) to be bigger than (𝔸_{f}^{(p)})^{×} (resp. ℚ_{ℓ}^{×}). The phenomenon of having large stabilizer subgroups occurs only for points which are somewhat special. First, points of A_{g}^{∗} at the boundary tend to have large stabilizer subgroups. For instance the 0-dimensional cusp “\sqrt{−1}∞ · I_{g}” contains a maximal parabolic subgroup of GSp_{2g}(ℚ_{ℓ}) with Levi factor GL_{g}(ℚ_{ℓ}) , even with level structure thrown in. For a point y in the interior, having a large stabilizer subgroup means that the abelian variety A_{y} has more endomorphisms than it is entitled to. This happens for example if y is rational over a finite field. In this case A_{y} has sufficiently many complex multiplication, namely End_{k}(A_{y}) ⊗_{ℤ} ℚ contains a commutative semisimple algebra of dimension 2g over ℚ. An extreme case is when A_{y} is supersingular, in which case we have End_{k}(A_{y}) ⊗_{ℤ} ℚ_{ℓ} ≅ M_{2g×2g}(ℚ_{ℓ}).

We shall prove at the end of this section that for a point y ∈ A_{g}(k), its primeto-p Hecke orbit G^{(p)}(y) (resp. ℓ-power Hecke orbit G_{ℓ}(y) ) is a finite set if and only if A_{y} is supersingular. Combined with the computation at the boundary of A_{g} and the results in [EO], this implies that Z_{ℓ} always contains supersingular points for every prime number ℓ ≠ p. More precisely, in [EO] it is shown that A_{g} has a natural stratification (called “canonical stratification” in loc. cit.), such that each stratum is quasi-affine and stable under all prime-to-p Hecke correspondences. The open stratum corresponds to ordinary points, and the 0-dimensional stratum corresponds to superspecial points. If Z_{ℓ}^{∗} meets the 0-dimensional cusp of A_{g}^{∗}, the calculation in §2 implies that Z_{ℓ} = A_{g}. This calculation can be generalized to other cusps at the boundary of A_{g}, and an induction on g gives Z_{ℓ} = A_{g} if Z_{ℓ}^{∗} meets the boundary of A_{g}. So we may assume that Z_{ℓ} is proper over Spec(k) . For any stratum S of A_{g}, unless the intersection of Z_{ℓ} with S is finite, Z_{ℓ} will have to contain a point in a lower stratum. This argument can be repeated, so eventually there exists a stratum S such that S ∩ Z_{ℓ} is non-empty and 0-dimensional. Then every point in S ∩ Z_{ℓ} has finite ℓ-power Hecke orbit, therefore is supersingular. Applying our general principle, one sees that an affirmative answer to the question (Q 2) implies that Z_{ℓ} is equal to A_{g} for every prime number ℓ ≠ p. Since (Q 2) turns out to be difficult, instead we proceed with a sequence of applications of the general method of using points with large stabilizer subgroups.

Let Z be the Zariski closure of the prime-to-p Hecke orbit of an ordinary point of A_{g}(k) as before, and let Z^{∗} be the Zariski closure of Z in A_{g}^{∗}. The logical structure of our proof is as follows:

Step 1. Show that if Z^{∗} meets the 0-dimensional cusp of A_{g}, then Z = A_{g}.

The proof of step 1 consists of a direct computation. This is possible because the structure of the completed local ring at the 0-dimensional cusp of A_{g}^{∗} is known, and the action of the stabilizer subgroup on the completed local ring is given by classical formulas. This step is explained in §2.

Step 2. Reduction of (Q 1) to its analogue for the Hilbert-Blumenthal moduli spaces.

Since Z contains ordinary points of A_{g} which are defined over finite fields, one may assume that A_{x} is defined over a finite field 𝔽_{q}. To simplify the exposition, let us assume that all simple factors of A_{x} are isogenous. Choose a suitable totally real subfield F ⊆ End_{k}(A_{x}) ⊗_{ℤ} ℚ which is fixed by the Rosati involution such that [F : ℚ] = dim(A) . Such an F always exists, see Lemma 4 in section 3. To further simplify the exposition, we assume that O_{F} ⊆ End_{k}(A_{x}) and that the principal polarization of A is O_{F} -linear. Then there is a Hilbert-Blumenthal moduli space M_{F}, a morphism M_{F} → A_{g} sending the boundary of M_{F} to the 0dimensional cusp of A_{g}, and an ordinary point x̃ ∈ M_{F} (k) mapping to x ∈ A_{g}(k). The Hecke correspondences on M_{F} come from the reductive ℚ-algebraic group GL_{2}(F). Moreover the morphism M_{F} → A_{g} is compatible with the Hecke actions with respect to a homomorphism of groups GL(2, 𝔸_{f,F}^{(p)} )^{red} → GSp_{2g}(𝔸_{f,ℚ}^{(p)}) . Here

GL(2, 𝔸_{f,F}^{(p)} )^{red} is the group of all elements in GL(2, 𝔸_{f,F}^{(p)} ) with determinants in

𝔸_{f,ℚ}^{(p)×}. Since there are two different groups and Hecke correspondence coming from them, we introduce the following definition to avoid possible confusion. We shall call the Hecke orbit of a point x̃ in M_{F} for the group GL(2, 𝔸_{f,F}^{(p)} )^{red} the reduced prime-to-p F -Hecke orbit of x̃ in M_{F} . Similarly we define the group GL_{2}(ℚ_{ℓ})^{red} to be the group of all elements in GL_{2}(ℚ_{ℓ}) with determinant in ℚ_{ell}^{×}, and call the GL_{2}(ℚ_{ℓ})^{red}-Hecke orbit the reduced ℓ-power F -Hecke orbit. If we can prove that the Zariski closure of the reduced prime-to-p F -Hecke orbit of x̃ is equal to M_{F} , then Z^{∗} will contain the 0-dimensional cusp of A_{g}. Hence Z = A_{g} by Step 1. This step is explained in §3.

Now let x̃ be an ordinary point in M_{F} (\overline{𝔽}_{p}) and let Z̃ be the Zariski closure of the reduced prime-to-p F -Hecke orbit of x̃ in M_{F}. We must show that Z̃ = M_{F}.

Step 3. Use Serre-Tate coordinates at ordinary points of Z̃(\overline{𝔽}_{p}) to limit Z̃ to a finite number of possibilities.

For any smooth ordinary point ỹ ∈ M_{F} (\overline{𝔽}_{p}), the formal subscheme Z̃∧ ; ỹ of M_{F,ỹ}^{∧} is stable under the action of the stabilizer subgroup of ỹ, which contains primeto-p isogenies coming from a totally imaginary quadratic extension of F because Aỹ is defined over a finite field. Using the Serre-Tate coordinates on M_{F,ỹ}, the computation in §4 shows that there are only a finite number of possibilities for Z̃∧ ; ỹ. In any case Z̃∧ ; ỹ must be a “coordinate subspace” of M_{F,ỹ}, determined by a subset of the set of all prime ideals in O_{F} lying above p. By faithfully flat descent, this result can be globalized to a statement about tangent spaces of the smooth ordinary locus of Z̃.

Step 4. Examine the action of the stabilizer subgroup on the completed local ring of supersingular points of M_{F} to eliminate all other possibilities, and conclude that Z̃ = M_{F}.

If the closure of Z̃ in M_{F}^{∗} meets the boundary of M_{F}^{∗} , a similar but simpler calculation as in Step 1 shows that Z̃ = M_{F} . Otherwise Z̃ is proper over Spec(k), and the results in [E-O] implies that Z̃ contains supersingular points by an argument similar to one used before. The point is that results in Step 3 allows us to conclude that Z̃ = M_{F} from very weak information about the deformation theory at supersingular points. In fact the deformation space has a natural product structure, and all we need is that none of the projections of the formal completion of Z̃ to the individual factors is fixed under the action of the stabilizer subgroup.

Step 5. Examine the action of the stabilizer subgroup of supersingular points of M_{F} on the formal completion of M_{F} to show that the Zariski closure of the reduced ℓ-power F -Hecke orbit of an ordinary point in M_{F} is actually stable under all reduced prime-to-p F -Hecke correspondences ‘at supersingular points’. This forces the reduced ℓ-power F -Hecke orbit of any ordinary point of M_{F} to be dense by what we already know.

To conclude this section, we classify polarized abelian varieties over k with finite prime-to-p Hecke orbits. Let (A_{0}, λ_{0}) be a polarized abelian variety over k of dimension g ≥ 1. Recall that k is algebraically closed of characteristic p > 0. Let A_{0}^{t} be the dual abelian variety of A_{0}. For any prime ℓ′ different from p, let

H_{1}(A_{0}, ℤ_{ℓ′}) = H_{ét}^{1}(A_{0}^{t} , ℤ_{ℓ′}(1) ) ,

a free ℤ′_{ℓ}-module of rank 2g. For ℓ′ = p, let

H_{1}(A_{0}, ℤ_{p}) = H_{crys}^{1}(A_{0}^{t}/W (k) , O_{A_{0}^{t}/W (k)} ) ⊗_{W (k)} ℤ_{p}(1) ,

a free W (k)-module of rank 2g with a natural F -crystal structure. Here ℤ_{p}(m) denotes H_{crys}^{2}(ℙ_{k}^{1}/W (k) , O_{ℙ_{k}^{1}/W (k)} )^{⊗(−m)} for any m ∈ ℤ. Let

H_{1}(A_{0}, ℚ_{ℓ′}) = H_{1}(A_{0}, ℤ_{ℓ′}) ⊗_{ℤ_{ℓ′}} ℚ_{ℓ′} , if ℓ′ ≠ p ;
H_{1}(A_{0}, ℚ_{p}) = H_{1}(A_{0}, ℤ_{p}) ⊗_{W (k)} B(k) , where B(k) = frac(W (k)) .

For any prime ℓ′ the polarization λ_{0} induces a ℤ_{ℓ′}-linear map

λ_{0,ℓ′} : H_{1}(A_{0}, ℤ_{ℓ′}) ⟶^{∼} H_{1}(A_{0}^{t}, ℤ_{ℓ′}) ,

which becomes an isomorphism when tensored with ℚ_{ℓ′} . The Poincaré sheaf on the product A_{0} ×_{Spec(k)} A_{0}^{t} induces the nondegenerate Weil pairing

H_{1}(A_{0}, ℤ_{ℓ′}) ⊗ H_{1}(A_{0}^{t}, ℤ_{ℓ′}) ⟶ ℤ_{ℓ′}(1) .

The Weil pairing and the polarization together give a pairing

λ_{0,ℓ′} : ⋀^{2} H_{1}(A_{0}, ℤ_{ℓ′}) → ℤ_{ℓ′}(1) ,

which is nondegenerate after tensoring with ℚ_{ℓ′} . For any prime ℓ′ the endomorphism ring End_{k}(A_{0}) (resp. End_{k}(A_{0}) ⊗_{ℤ} ℚ) operates on H_{1}(A_{0}, ℤ_{ℓ′}) (resp. H_{1}(A_{0}, ℚ_{ℓ′})) by functoriality, and the action extends by linearity to End_{k}(A_{0})⊗_{ℤ}ℤ_{ℓ′} (resp. End_{k}(A_{0})⊗_{ℤ}ℚ_{ℓ′}). The algebra End_{k}(A_{0})⊗_{ℤ}ℚ is a finite dimensional semisimple algebra over ℚ with a positive Rosati involution ∗ given by the polarization λ_{0}. This semisimple algebra with involution (End_{k}(A_{0})⊗_{ℤ} ℚ, ∗) defines a reductive linear algebraic group H over ℚ whose ℚ-rational points H(ℚ) consists of all elements h ∈ End_{k}(A_{0}) ⊗_{ℤ} ℚ such that h^{∗} · h = h · h^{∗} = c(h) · id for some c(h) ∈ ℚ^{×}. The lattice H_{1}(A_{0}, ℤ_{ℓ′}) ⊆ H_{1}(A_{0}, ℚ_{ℓ′}) defines a compact subgroup K_{ℓ′} ⊆ H(ℚ_{ℓ′}) consisting of all elements h ∈ H(ℚ_{ℓ′}) such that h · H_{1}(A_{0}, ℤ_{ℓ′}) = H_{1}(A_{0}, ℤ_{ℓ′}). K_{ℓ′} is a hyperspecial maximal compact subgroup of H(ℚ_{ℓ′}) for all but finitely many primes ℓ′. For ℓ′ different from p let G_{ℓ′} denotes the group of all ℚ_{ℓ′}-linear automorphisms of H_{1}(A_{0}, ℚ_{ℓ′}) which preserve the ℚ_{ℓ′}-polarization λ_{0,ℓ′} on H_{1}(A_{0}, ℚ_{ℓ′}) up to ℚ_{ℓ′}^{×}. The group G_{ℓ′} actually comes from a unique reductive linear algebraic group 𝔊_{ℓ′} over ℚ_{ℓ′} such that 𝔊_{ℓ′}(ℚ_{ℓ′}) = G_{ℓ′} . We have a natural inclusion H(ℚ_{ℓ′}) ⊆ G_{ℓ′}, induced by a natural inclusion H ×_{Spec ℚ} Spec ℚ_{ℓ′} ⊆ 𝔊_{ℓ′} .

Using (A_{0}, λ_{0}) as a base point, the non-p Hecke orbit of A_{0} is parameterized by the double coset

(H(ℚ) ∩ K_{p}) \ ∏_{ℓ′≠p}′ G_{ℓ′} / ∏_{ℓ′≠p} K_{ℓ′} .

This is standard: The set

∏_{ℓ′≠p}′ G_{ℓ′} / ∏_{ℓ′≠p} K_{ℓ′}

parameterizes the set of isomorphism classes of triples (β, A_{x} , λ_{x}) , where (A_{x}, λ_{x}) is a principally polarized abelian variety, and β : A_{x} → A_{0} is a prime-to-p quasiisogeny such that

β^{∗}(λ_{0}) = \frac{a}{b} λ_{x}

for some positive integers a, b which are prime to p. If (β_{1}, A_{1}, λ_{1}) and (β_{2}, A_{2}, λ_{2}) are two such triples, and

ξ : (A_{1}, λ_{1}) ⟶^{∼} (A_{2}, λ_{2})

is an isomorphism, then ξ determines an element h ∈ H(ℚ) such that h ∘ β_{1} = β_{2} ∘ ξ. This defines a map from the prime-to-p Hecke orbit G^{(p)}(A_{0}, λ_{0}) of A_{0} to the double coset

H(ℚ) ∩ K_{p} \ ∏_{ℓ′≠p}′ G_{ℓ′} / ∏_{ℓ′≠p} K_{ℓ′} .

It is easy to see that this is indeed a bijection. Similarly the ℓ-power Hecke orbit G_{ℓ}(A_{0}, λ_{0}) of A_{0} is parameterized by

(H(ℚ) ∩ ∏_{ℓ′≠ℓ} K_{ℓ′}) \ G_{ℓ} / K_{ℓ} .

Notice that if H(ℚ_{ℓ}) = G_{ℓ} for one prime number ℓ ≠ p, then H(ℚ_{ℓ′}) = G_{ℓ′} for all ℓ′ different from p, and by the finiteness of generalized class numbers for reductive groups over global fields the ℓ-power Hecke orbit of A_{0} is finite.

Proposition 1. Assume that ℓ is a prime number which is different from p = char.(k). If the ℓ-power Hecke orbit G_{ℓ}(A_{0}, λ_{0}) of a polarized abelian variety (A_{o}, λ_{0}) over k is finite, and dim(A_{0}) ≥ 1 , then A_{0} is a supersingular abelian variety. Conversely if (A_{0}, λ_{0}) is a supersingular polarized abelian variety, then its prime-to-p Hecke orbit is finite.

Proof. Assume that A_{0} is supersingular. Then H(ℚ_{ℓ′}) = G(ℚ_{ℓ′}) for all ℓ′ ≠ p , because the dimension of the semisimple algebra End_{k}(A_{0}) ⊗_{ℤ} ℚ_{ℓ′} is equal to 4dim(A)^{2}, which is equal to the dimension of End_{ℚ_{ℓ′}} (H_{1}(A_{0}, ℚ_{ℓ′}) . The prime-to-p Hecke orbit of A_{0} is also finite, again by the finiteness of class numbers. In fact if A_{0} is a supersingular, then H is a form of GSp_{2g}, its derived group H^{der} is simply connected, and H/H^{der} ≅ 𝔾_{m}. Since H^{der}(ℝ) is compact, the class number in general is bigger than one. Information about the class numbers is available but is not needed here.

Conversely if the ℓ-power Hecke orbit of A_{0} is finite for a prime number ℓ ≠ p, then the homogeneous space G_{ℓ} / H(ℚ_{ℓ}) is compact. This implies that H ×_{Spec ℚ} Spec ℚ_{ℓ} = 𝔊_{ℓ} by Wang’s generalization of the Borel density theorem, therefore H(ℚ_{ℓ}) is to G_{ℓ}. For the reader’s convenience, we recall a special case of Wang’s theorem: If G is a connected semisimple algebraic group defined over a nondiscrete locally compact field K without nontrivial K-anisotropic factors, and if Γ ⊂ G(K) is a closed subgroup in the Hausdorff topology such that G(K)/Γ has a finite G(K)-invariant measure, then Γ is Zariski dense in G. See [Wa], Cor. 1.4.

Clearly H(ℚ_{ℓ}) ⊆ G_{ℓ} is induced by (End_{k}(A_{0})⊗_{ℤ}ℚ_{ℓ} , ∗) ⊆ (End_{ℚ_{ℓ}}(H_{1}(A_{0} , ℚ_{ℓ})) , ∗) . Since the representation of G_{ℓ} on H_{1}(A_{0}, ℚ_{ℓ}) is absolutely irreducible, End_{ℚ_{ℓ}}(H_{1}(A_{0} , ℚ_{ℓ})) is the ℚ_{ℓ}-linear span of G_{ℓ}. It follows that

dim_{ℚ_{ℓ}}(End_{k}(A_{0}) ⊗_{ℤ} ℚ_{ℓ}) = dim_{ℚ_{ℓ}}(End_{ℚ_{ℓ}}(H_{1}(A_{0} , ℚ_{ℓ}))) = 4g^{2} ,

Hence dim_{ℚ}(End_{k}(A_{0}) ⊗_{ℤ} ℚ) = 4g^{2}. This implies that A_{0} is a supersingular abelian variety. Proposition 1 is proved. □

2. Calculation at the 0-dimensional cusp

Let Z be the Zariski closure of the prime-to-p Hecke orbit of an ordinary point x ∈ A_{g}(k) as in §1. In this section, we assume that the closure Z^{∗} of Z in the minimal compactification A_{g}^{∗} contains the 0-dimensional cusp of A_{g}. Therefore Z^{∗} is a Zariski closed subscheme of A_{g}^{∗}, stable under the ℓ-power Hecke correspondences for a prime number ℓ ≠ p, and contains the 0-dimensional cusp. In this section we shall show that these assumptions force Z to be equal to A_{g}.

The proof itself is an illustration of the general principle explained in §1. The point is that the the 0-dimensional cusp has a large “stabilizer subgroup” in the ℓ-power Hecke operators: The ℓ-power Hecke correspondences come from the group GSp_{2g}(ℚ_{ℓ}) , and the stabilizer subgroup of the 0-dimensional cusp contains a maximal parabolic subgroup P , which is a semidirect product of its Levi factor L and its unipotent radical U. The Levi component L is isomorphic to GL_{g}(ℚ_{ℓ}) ; the unipotent radical is commutative, and as a representation of GL_{g}(ℚ_{ℓ}) it is isomorphic to the second divided power product of the standard representation of GL_{g}(ℚ_{ℓ}). In this section we shall follow the notation in [FC]. The free abelian group X of rank g used in [FC], chapters 4 and 5 corresponds to the dual of the standard representation of our Levi factor GL_{g} here. Of course this depends on the choice of an isomorphism from the Levi factor to GL_{g}. In terms of the standard block form realization of Sp_{2g} and the maximal parabolic subgroup P which we use below, the isomorphism is induced by sending an upper-triangular block to the entry A in the upper-left corner of the block.

As before ℓ is a prime number different from p, where p > 0 is the characteristic of the algebraically closed field k. For any positive integer n prime to p and a choice of a primitive n-th root of unity, let A_{g,n} denote the moduli stack of principally polarized abelian varieties over k with symplectic level-n-structure; let A_{g,n}^{∗} denotes the minimal compactification of A_{g,n}. We choose a trivialization

e_{ℓ} : ℤ[1/ℓ]/ℤ ⟶^{∼} μ_{ℓ^{∞}}(k)

of μ_{ℓ^{∞}} over k. Then we can consider the projective system

\underleftarrow{lim}_{m∈ℕ} A_{g,ℓ^{m}} ,

the ℓ-adic tower of moduli spaces of principally polarized abelian varieties over k with symplectic ℓ-power level-structure. On each individual A_{g,ℓ^{m}}, the group

Sp_{2g}(ℤ/ℓ^{m}ℤ) = {γ ∈ M_{2g} (ℤ/ℓ^{m}ℤ) | ⟨γv, γw⟩ = ⟨v, w⟩ ∀v, w ∈ (ℤ/ℓ^{m}ℤ)^{2g} }

operates on A_{g,ℓ^{m}}/A_{g}. On the ℓ-adic tower, of course the group

Sp_{2g}(ℤ_{ℓ}) = \underleftarrow{lim} Sp_{2g}(ℤ/ℓ^{m}ℤ)

operates “as covering transformations over A_{g}”. Here and below ⟨ , ⟩ denotes the standard symplectic pairing on the 2g-dimensional column vectors. In other words,

⟨( \begin{matrix}a_{1} \\ b_{1} \end{matrix} ) , ( \begin{matrix}a_{2} \\ b_{2} \end{matrix} )⟩ = ^{t}a_{1} · b_{2} − ^{t}b_{1} · a_{2} ∀a_{1} , b_{1} , a_{2} , b_{2} ∈ R^{g} ,

where R is any commutative ring, and R^{g} denotes the space of g-dimensional column vectors with entries in R. Moreover, this action of Sp_{2g}(ℤ_{ℓ}) extends to an action of the much bigger group

GSp_{2g}(ℚ_{ℓ}) = \{γ ∈ M_{2g} (ℚ_{ℓ}) | \begin{matrix}∃ c(γ) ∈ ℚ_{ℓ}^{×} such that \\ ⟨γv, γw⟩ = c(γ)⟨v, w⟩ ∀v, w ∈ ℚ_{ℓ}^{2g} \end{matrix} \}

on the tower \underleftarrow{lim} A_{g,ℓ^{m}} . This action of GSp_{2g}(ℚ_{ℓ}) on the ℓ-adic tower \underleftarrow{lim} A_{g,ℓ^{m}} gives the ℓ-power Hecke correspondences on A_{g}.

For our purpose it is convenient to use the following subgroup

G_{ℓ} = {γ ∈ M_{2g} (ℤ[1/ℓ]) | ∃ c(γ) ∈ ℓ^{ℤ} s.t. ⟨γv, γw⟩ = c(γ)⟨v, w⟩ ∀v, w ∈ ℤ[1/ℓ]^{2g}}

of GSp_{2g}(ℚ_{ℓ}) when considering ℓ-power Hecke operators. The assumption on Z means that the inverse image of Z in the ℓ-power tower \underleftarrow{lim} A_{g,ℓ^{m}} is stable under G_{ℓ}. Consequently the inverse image of Z^{∗} in \underleftarrow{lim} A_{g,ℓ^{m}}^{∗} is also stable under the action of G_{ℓ}.

Recall that our goal is to deduce Z = A_{g} from the hypothesis that Z is a Zariski closed subscheme of A_{g} over k which is stable under the ℓ-power Hecke correspondences and such that the Zariski closure Z^{∗} in A_{g}^{∗} contains the 0-dimensional cusp. This implies that Z contains a nonempty open subscheme whose geometric points are all ordinary. Since the problem is trivial for g ≤ 1, we may assume that g ≥ 2.

If m′ > m ≥ 0, the natural morphism

A_{g,ℓ^{m′}}^{∗} ⟶ A_{g,ℓ^{m}}^{∗}

sends the standard 0-dimensional cusp of A_{g,ℓ^{m′}}^{∗} to the standard 0-dimensional cusp of A_{g,ℓ^{m}}^{∗}. If we denote by Spf (R_{ℓ^{m}}) the formal completion of A_{g,ℓ^{m}}^{∗} at the standard 0-dimensional cusp, then we have a projective system

\underleftarrow{lim}_{m} Spf (R_{ℓ^{m}}) .

The inverse image of Z^{∗} in Spf (R_{ℓ^{m}}) is a closed formal subscheme Spf (R_{ℓ^{m}}/I_{ℓ^{m}}), and the projective limit

\underleftarrow{lim}_{m} Spf (R_{ℓ^{m}}/I_{ℓ^{m}}) ⊆ \underleftarrow{lim}_{m} Spf (R_{ℓ^{m}})

is stable under the stabilizer subgroup in G_{ℓ} for the projective limit of the standard 0-dimensional cusp.

Some explanation is in order. First we clarify what we meant by “the standard 0-dimensional cusp”. In [FC], the minimal compactification A_{g,ℓ^{m}}^{∗} of A_{g,ℓ^{m}} is constructed by blowing down any smooth arithmetic toroidal compactification \overline{A}_{g,ℓ^{m}} using sections of powers of the Hodge line bundle \underline{ω}. We follow the notation in [FC] Chapter 4. Let X = ℤ^{g}, X_{ℓ^{m}} = Y_{ℓ^{m}} = (1/ℓ^{m}) X. Let C = C(X) be the convex cone of all positive semi-definite symmetric bilinear forms on X_{ℝ} whose radicals are defined over ℚ. We choose a smooth GL(X)-admissible polyhedral cone decomposition {σ_{α}}_{α∈J} of C(X). The part of the toroidal compactification given by the cone decomposition {σ_{α}}_{α∈J} of C(X) over the standard standard 0-dimensional cusp can be described as follows. Give C(X) the integral structure attached to the lattice of integral valued symmetric bilinear forms on (ℓ^{m}X) × (ℓ^{m}X) . Then over

Spec k [X_{ℓ^{m}} ⊗ X_{ℓ^{m}}/μ ⊗ ν − ν ⊗ μ]_{μ,ν∈X_{ℓ^{m}}}

we have a universal bimultiplicative function

b_{ℓ^{m}} : Y_{ℓ^{m}} × X_{ℓ^{m}} → (k [X_{ℓ^{M}} ⊗ X_{ℓ^{m}}/μ ⊗ ν − ν ⊗ μ]_{μ,ν∈X_{ℓ^{m}}} )^{×} .

Now the construction in [FC] Chapter 4, §§3, 6 gives us a semiabelian scheme

♡G → \overline{E}^{∧}/ GL(X)(ℓ^{m})

over the GL(X)(ℓ^{m})-quotient of the formal completion of the torus embedding E determined by the cone decomposition {σ_{α}}_{α∈J} with integral structure given by (1/ℓ^{m}) Γ_{2}(X^{∗}) , along the union of strata corresponding to those cones lying in the interior of C. Here Γ_{2}(X^{∗}) denotes the second divided power product of X^{∗}. The generic fiber of the semiabelian scheme ^{♡}G → \overline{E}^{∧}/ GL(X)(ℓ^{m}) is abelian, and the abelian part has a symplectic level-ℓ^{m}-structure. The level-structure comes from the natural identification of Y_{ℓ^{m}}/Y with (ℤ/ℓ^{m}ℤ)^{g} and our choice of ℓ^{m}-th root of unity e_{ℓ}(1/ℓ^{m}). Roughly, b_{ℓ^{m}} determines a map from Y_{ℓ^{m}} = X_{ℓ^{m}} to G̃_{ℓ^{m}}(K), where K is the fraction field of the ring

k [X_{ℓ^{M}} ⊗ X_{ℓ^{m}}/μ ⊗ ν − ν ⊗ μ]_{μ,ν∈X_{ℓ^{m}}}

and G̃_{ℓ^{m}} is the split torus with character group X_{ℓ^{m}}. On G_{ℓ^{m}} = “G̃_{ℓ^{m}}/Y ”, we have a polarization of degree ℓ^{2mg} which is the pull-back from the principal polarization on “G̃_{ℓ^{m}}/Y_{ℓ^{m}}”. This polarization descends to a principal polarization on the quotient of G_{ℓ^{m}} by the ℓ^{m}-torsion points of its multiplicative part. The abelian part of this quotient has a natural level-ℓ^{m}-structure. This semiabelian scheme, together with its principal polarization and level-ℓ^{m}-structure on the abelian part, descends first to \overline{E}^{∧} and then to \overline{E}^{∧}/ GL(X)(ℓ^{m}) . This gives the semiabelian scheme ^{♡}G → \overline{E}^{∧}/ GL(X)(ℓ^{m}) above, together with its principal polarization and level-ℓ^{m}-structure on the abelian part.

The rings R_{ℓ^{m}}’s are calculated in [FC], Chapter 5, §2: There is a natural isomorphism

R_{ℓ^{m}} ≅ k[[q^{λ}]]_{λ≥0}^{Γ_{ℓ^{m}}} ,

the ring of Γ_{ℓ^{m}}-invariants in k[[q^{λ}]]_{λ≥0} , where λ runs through all positive semidefinite elements in (1/ℓ^{m}) · S^{2}(X) (i.e. elements in (1/ℓ^{m}) · S^{2}(X) having non-negative values on C(X) ) and Γ_{ℓ^{m}} is the principal congruence subgroup GL(X)(ℓ^{m}) of GL(X) of level ℓ^{m}. Taking the inverse limit, we get the ring

R = R(ℓ) =^{def} ⋃_{n} R_{ℓ^{m}} .

Our previous discussion tells us that inside the ring R we have an ideal

I = I(ℓ) =^{def} ⋃_{n} I_{ℓ^{m}} .

which is stable under the the stabilizer subgroup in G_{ℓ} for the projective limit of the standard 0-dimensional cusp.

This stabilizer subgroup P_{ℓ} in G_{ℓ} for the standard 0-dimensional cusp of \underleftarrow{lim} A_{g,ℓ^{m}} can be explicitly described in block form; it is

P_{ℓ} =^{def} \{( \begin{matrix}A & B \\ 0 & D \end{matrix} ) | A, B, D ∈ M_{g}(ℤ[1/ℓ]), A ·^{t}D ∈ ℓ^{ℤ} · Id, A ·^{t}B = B ·^{t}A\} .

We shall prove that if I ≠ (0), then Spf(R/I) is contained in the boundary. Since the minimal compactification A_{g,ℓ^{m}}^{∗} has nasty singularities at the boundary, it is not so easy to prove this directly. Instead we prove that the inverse image of Spf(R/I) in a toroidal compactification is contained in the boundary. In other words we use the toroidal desingularization of the minimal compactification A_{g,ℓ^{m}}^{∗} to handle the singularity.

The group P_{ℓ} operates on the rings R_{ℓ^{m}} by pulling back formal functions. We shall only need its intersection with Sp_{2g}(ℚ_{ℓ}), namely the subgroup

P′_{ℓ} =^{def} \{( \begin{matrix}A & B \\ 0 & D \end{matrix} ) | A, B, D ∈ M_{g}(ℤ[1/ℓ]), A · ^{t}D = Id, A ·^{t}B = B · ^{t}A\} .

When the formal functions are represented as formal power series as above, the action of the elements of P′_{ℓ} is given by classical formulas. If f = ∑_{λ} a_{λ} · q^{λ} is an element of R, then the pull-back of f by an element of P′_{ℓ} is given by

( \begin{matrix}A & B \\ 0 & D \end{matrix} ) : f = ∑_{λ} a_{λ} · q^{λ} ⟼ ∑_{λ} e ((λ , B · ^{t}A)) a_{λ} q^{λ∗A} .

The expressions (λ, B · ^{t}A) and λ ∗ A have to be explained. We identify the second divided power product Γ_{2}(X^{∗}) of X^{∗} with the group of all g ×g symmetric matrices with integer entries; similarly after change of rings. In the formula above (λ , B ·^{t}A) is the value of the pairing between λ ∈ S^{2}(X ⊗ ℤ[1/ℓ]) and the symmetric matrix

B · ^{t}A ∈ M_{g×g}(ℤ[1/ℓ]) = Γ_{2}(X^{∗} ⊗ ℤ[1/ℓ]) .

In the traditional treatment of Siegel modular forms S^{2}(X) is identified with the group of all g × g half-integral matrices with integral diagonal entries, Γ_{2}(X^{∗}) is identified with the group of all g × g symmetric integral matrices, and the pairing between them becomes the composition of the matrix product with the trace function. As to the expression λ ∗ A, recall our decree that X^{∗} corresponds to the standard representation of GL_{g}(ℤ). The expression λ∗A in the formula above is the image of λ under the transpose of the action of A on Γ_{2}(X^{∗}). In classical notation, if λ is given by a symmetric even half-integral matrix L as we explained above, then λ ∗ A corresponds to the symmetric matrix ^{t}A L A.

As P′_{ℓ} is a semi-direct product of its unipotent radical U_{ℓ} (those with A = D = Id) and its Levi factor L_{ℓ} (those with B = 0), we also give the formula for the action of these two subgroups.

U_{ℓ} = \{( \begin{matrix}I_{g} & B \\ 0 & I_{g} \end{matrix} ) | B ∈ M_{g}(ℤ[1/ℓ]), B = ^{t}B\} ;

U_{ℓ} ∋ ( \begin{matrix}I_{g} & B \\ 0 & I_{g} \end{matrix} ) : f = ∑_{λ} a_{λ} · q^{λ} ⟼ ∑_{λ} e ((λ , B)) a_{λ} q^{λ} , f ∈ R .

L_{ℓ} = \{( \begin{matrix}A & 0 \\ 0 & D \end{matrix} ) | A, D ∈ M_{g}(ℤ[1/ℓ]), A · ^{t}D = Id\} ;

L_{ℓ} ∋ ( \begin{matrix}A & 0 \\ 0 & D \end{matrix} ) : f = ∑_{λ} a_{λ} · q^{λ} ⟼ ∑_{λ} a_{λ} q^{λ∗A} , f ∈ R .

We are in a position to state the main result of this section.

Proposition 2. If (0) ≠ I_{ℓ^{m}} is an ideal in R_{ℓ^{m}} such that I_{ℓ^{m}}R is stable under the action of P′_{ℓ}. Then I_{ℓ^{m}} contains a power of the maximal ideal of R_{ℓ^{m}} . Consequently, if Z is a closed subscheme stable under the ℓ-power Hecke correspondences, and if the Zariski closure Z^{∗} of Z in A_{g}^{∗} contains the 0-dimensional cusp of A_{g}^{∗}, then Z = A_{g}.

Before giving the proof, we would like to explain the idea and outline the proof. We have explicit formulas for the formal completion of A_{g,ℓ^{m}}^{∗} at the standard 0-dimensional cusp (i.e. Spf (R_{ℓ^{m}}) ), the stabilizer subgroup of the standard 0dimensional cusp in the projective system \underleftarrow{lim} Spf (R_{ℓ^{m}}) , and the action of the stabilizer subgroup on \underleftarrow{lim} Spf (R_{ℓ^{m}}) . Thus it seems reasonable that one should be able to prove Proposition 2 by direct computation. This is only partly true, for the rings R_{ℓ^{m}} are difficult. But the toroidal resolution comes to our rescue.

Let \overline{A}_{g,ℓ^{m}} be the toroidal compactification over k given by a chosen smooth cone decomposition \{σ_{α}\}_{α∈J} of C(X). The inverse image in \overline{A}_{g,ℓ^{m}} of the standard 0dimensional cusp has a stratification ⨆_{α} W_{α} with the strata parameterized by

{α | σ_{α} ⊂ C(X)^{∘}} / GL(X)(ℓ^{m}) .

We assume that ℓ^{m} ≥ 3 for convenience. The formal completion of \overline{A}_{g,ℓ^{m}} along the inverse image of the standard 0-dimensional cusp is covered by affine open formal subschemes S_{α}, again parameterized by {α | σ_{α} ⊂ C(X)^{∘}} / GL(X)(ℓ^{m}) . Let σ_{α} ⊂ C(X)^{∘} be such a cone. The coordinate ring of S_{α} is naturally isomorphic to the formal completion of

⨁_{{λ ∈ \frac{1}{ℓ^{m}} S^{2}(X) \atop λ ≥ 0 on σ}} k · q^{λ}

along the ideal

⨁_{{λ ∈ \frac{1}{ℓ^{m}} S^{2}(X) \atop λ > 0 on \overline{σ} ∩ C(X)^{∘}}} k · q^{λ} .

Call this completed ring R_{σ,ℓ^{m}} and let J_{σ,ℓ^{m}} be the ideal in R_{σ,ℓ^{m}} generated by the ideal above. Since the cone compactification σ_{α} ⊂ C(X)^{∘} is smooth, J_{σ,ℓ^{m}} is a principal ideal.

If

f = ∑_{λ} a_{λ} q^{λ} ∈ R_{σ,ℓ^{m}}

is an element of R_{σ,ℓ^{m}} such that there exists a λ_{1} with the following properties (a) a_{λ_{1}} ≠ 0 .

(b) λ ∈ λ_{1} · J_{σ,ℓ^{m}} for any λ with a_{λ} ≠ 0 . Equivalently, λ − λ_{1} > 0 on every face τ of σ such that \overline{τ} ⊂ C(X)^{∘} ∪ \{0\} .

Then we say that f has a_{λ_{1}} q^{λ_{1}} as the leading term, with respect to the ideal J_{σ,ℓ^{m}} . The condition (b) means that f is congruent to a_{λ_{1}} q^{λ_{1}} modulo 1 + J_{σ,ℓ^{m}} in the multiplicative sense. Clearly if f has leading term a_{λ_{1}} q^{λ_{1}} with a_{λ_{1}} ≠ 0 , then f · R_{σ,ℓ^{m}} = q^{λ_{1}} · R_{σ,ℓ^{m}} as ideals of R_{σ,ℓ^{m}} .

From the above discussion, we see that if an ideal Q of R_{ℓ^{m}} contains an element f_{σ} with a leading term a_{λ_{σ}} q^{λ_{σ}} with respect to J_{σ,ℓ^{m}} for every cone σ in a system of representatives of

{σ_{α}}_{α∈J} / GL(X)(ℓ^{m}) ,

then Spf (R_{ℓ^{m}} / Q) is contained in the formal completion of the boundary of A_{g,ℓ^{m}}^{∗} at the standard 0-dimensional cusp. This follows from Grothendieck’s formal function theorem, applied to the morphism from \overline{A}_{g,ℓ^{m}} → A_{g,ℓ^{m}}^{∗} over the standard 0-dimensional cusp.

Similarly, if Q is a finitely generated ideal of R such that for every cone σ in the cone decomposition {σ_{α}}_{α∈J} , Q contains an element f_{σ} which has a leading term a_{λ_{σ}} q^{λ_{σ}} with respect to the ideal

J_{σ} = ∪_{n} J_{σ,ℓ^{m}} .

Then Spf (R / Q) is contained in the the boundary of \underleftarrow{lim} A_{g,ℓ^{m}}^{∗ ∧} , the projective limit of the completion of the boundary of A_{g,ℓ^{m}}^{∗} at the standard 0-dimensional cusp.

We have to produce sufficiently many elements in the ideal I ⊂ R which have leading terms. In order to get elements with leading terms, one only has to “shave off” undesirable terms in the Fourier expansion of any given nonzero element in I. For this we use the action of U_{ℓ}. This allows us to “shave off” finitely many terms in the Fourier expansion of a given non-zero element of I. Then we show that for cones σ in the cone decomposition such that \overline{σ} ⊂ C(X)^{∘} ∪ \{0\}, just throwing away a finite number of terms is enough to produce elements with leading terms. For cones “touching the boundary of C(X)”, things get a little more complicated, but the same idea works. We formulate these parts as separate lemmas:

Lemma 1. For any nonzero element

0 ≠ f = ∑_{λ} a_{λ} · q^{λ} ∈ I

in I, and for any finite subset {λ_{0}, λ_{1}, …, λ_{N}} ⊆ support(f) , there exists an element g ∈ I such that

λ_{0} ∈ support(g) ⊆ support(f) − {λ_{1}, …, λ_{N}} .

Here support(f) denotes the set of all λ’s such that a_{λ} ≠ 0, similarly for support(g) . In other words, g = ∑_{λ} b_{λ} · q^{λ} is such that b_{λ_{0}} ≠ 0, b_{λ_{1}} = … = b_{λ_{N}} = 0, and b_{λ} ≠ 0 ⇒ a_{λ} ≠ 0.

Lemma 1 is an immediate consequence of the formula for the U_{ℓ}-action and the linear independence of characters of an abelian group. In fact we can choose g to be a suitable linear combination of elements in the U_{ℓ}-orbit of f.

Lemma 2. Let b ∈ C(X)^{∘} be a positive definite symmetric bilinear form on X_{ℝ}. Let M be a given positive real number. Then there are only finitely many integral elements λ in the dual cone of positive semidefinite elements in S^{2}(X_{ℝ}) such that (λ, b) ≤ M .

Lemma 2 is certainly well-known, and it holds in the more general situation of the polyhedral reduction theory for self-adjoint cones as in [AMRT], Chap. 2. We supply a direct elementary proof for the convenience of the reader. We shall identify S^{2}(X) with the group of all half-integral symmetric g × g matrices with integral diagonal entries, Γ_{2}(X^{∗}) with symmetric integral g × g matrices, and the pairing between them is the composition of the matrix product and the trace function. The element b ∈ C(X)^{∘} corresponds to a positive definite symmetric g × g matrix B. Clearly there exists a positive constant c > 0 such that c · B ≫ Id, in other words c·B dominates Id as positive definite symmetric g×g matrices. Therefore if L is an half-integral positive semi-definite symmetric matrix with integral diagonal entries such that tr(L · B) ≤ M, then the absolute values of the diagonal entries of L are all bounded by cM. It follows that the absolute values of all off-diagonal entries of L are also bounded by cM. Therefore there are only finitely many possible L’s. This proves Lemma 2.

Proof of Proposition 2. Of course the key is to produce enough elements in the ideal I_{ℓ^{m}}R with leading terms. Since I_{ℓ^{m}} ≠ (0) there exists a nonzero function f = ∑_{λ} a_{λ} q^{λ} in the ideal I_{ℓ^{m}} . Let σ be a cone in the cone decomposition {σ_{α}} such that \overline{σ} ⊂ C(X)^{∘} ∪ \{0\}. Since f ≠ 0 and I_{ℓ^{m}} ≠ (0) , there exists a λ_{0} such that a_{λ_{0}} ≠ 0 and λ_{0} ≠ 0. By Lemma 2, there exists a finite set Λ ⊂ support(f) such that λ − λ_{0} > 0 on \overline{σ} − \{0\}, for every λ ∈ support(f) − Λ . By Lemma 1, there exists an element f_{1} ∈ I with leading term q^{λ_{0}}.

We only have to generalize the above argument a little in order to deal with the the general case of a cone σ ⊂ C(X)^{∘} in the cone decomposition. Again let f = ∑_{λ} a_{λ} q^{λ} ∈ R_{ℓ^{m}} be a nonzero function in the ideal I_{ℓ^{m}} . Using Lemma 2 and Lemma 1, for any positive integer N > 0, we can find a λ_{0} such that a_{λ_{0}} ≠ 0 , and a suitable linear combination g_{N} of elements in the U_{ℓ}-orbit of f such that

g_{N} ∈ q^{λ_{0}} + J_{σ,ℓ^{m}}^{N} ⊂ R_{ℓ^{m}} .

The point is that whether a monomial q^{λ} with λ ∈ (1/ℓ^{m})S^{2}(X) is in J_{σ,ℓ^{m}}^{N} can be detected by the ℝ-valued function given by the restriction of λ to the largest closed face \overline{τ} of \overline{σ} such that \overline{τ} ⊂ C(X)^{∘} ∪ \{0\}. Also, g_{N} ∈ R_{ℓ^{m}} because support( g_{N}) ⊆ support(f) . Since g_{N} ∈ I_{ℓ^{m}}R ⊂ I_{ℓ^{m}}R_{σ} := ⋃_{n} R_{σ,ℓ^{n}} , and ⋃_{n} R_{σ,ℓ^{n}} is faithfully flat over R_{σ,ℓ^{m}} , g_{N} ∈ I_{ℓ^{m}}R_{σ,ℓ^{m}} . Thus we see that q^{λ_{0}} ∈ IR_{σ,ℓ^{m}} + J_{σ,ℓ^{m}}^{N} for every N. This implies that q^{λ_{0}} ∈ IR_{σ,ℓ^{m}} since the ideal q^{λ_{0}} ∈ I_{ℓ^{m}}R_{σ,ℓ^{m}} is closed in the J_{σ,ℓ^{m}}-adic topology. So we have found an element in I_{ℓ^{m}}R_{σ,ℓ^{m}} with a leading term, for any cone σ ⊂ C(X)^{∘}. This proves Proposition 2. □

Notice that in the proof above we only needed the U_{ℓ}-action. Thus we have actually proved a strengthened form of Proposition 2:

Proposition 2′. If (0) ≠ I_{ℓ^{m}} is an ideal in R_{ℓ^{m}} such that I_{ℓ^{m}}R is stable under the action of U_{ℓ}. Then I_{ℓ^{m}} contains a power of the maximal ideal of R_{ℓ^{m}}.

3. Reduction to the Hilbert-Blumenthal case

In this section we reduce the question (Q 1) to an analogous question for the Hilbert-Blumenthal moduli spaces over k. The underlying idea is very simple. We want to show that every Zariski closed subscheme Z of A_{g} over Spec k which is stable under all prime-to-p Hecke correspondences and which contains ordinary points is necessarily equal to A_{g}. Since Z is of finite presentation over Spec k and A_{g} is defined over 𝔽_{p}, there exists a subring D ⊂ k of finite type over 𝔽_{p}, a scheme Z^{∗} proper and flat over D, a closed embedding Z_{/D}^{∗} ↪ A_{g}^{∗}_{/D} over D of Z^{∗} into the minimal compactification, such that every fiber of Z^{∗} → Spec D contains points corresponding to ordinary abelian varieties, and such that Z^{∗} ×_{Spec D} Spec k is equal to Z^{∗}, the Zariski closure of Z in A_{g}^{∗}. Since the generic fiber of Z^{∗} is stable under all prime-to-p Hecke correspondences, so is Z^{∗}. To show that Z is equal to A_{g}, it suffices to show that the fibers of Z^{∗} → Spec D over finite fields are equal to A_{g}^{∗}. Therefore we may assume without loss of generality that Z contains a point x_{0} defined over some finite field 𝔽_{q}. The point x_{0} corresponds to a principally polarized ordinary abelian variety (A_{0}, λ_{0}) defined over 𝔽_{q}. It is well-known that every simple abelian variety over a finite field has complex multiplication by a CM field of degree twice that of the dimension of the abelian variety, therefore has real multiplication by the totally real subfield of the CM field. A reference is [Ta]. In fact, the endomorphism algebra End(A_{0}) ⊗_{ℤ} ℚ of an absolutely simple ordinary abelian variety A_{0} defined over a finite field is a CM field, generated by a Frobenius element. Therefore if A_{0} is absolutely simple, then End_{k}(A_{0}) is an order in a CM field L. Let F ⊂ L be the totally real subfield of L. If in addition the ring of integers O_{F} in F is contained in End_{k}(A_{0}), x_{0} is in the image of a morphism from a Hilbert-Blumenthal moduli scheme M_{F} to A_{g} over k. Moreover, this morphism is compatible with Hecke operators. Hence if we can prove that the reduced prime-to-p F -Hecke orbit of [A_{0}] in M_{F} is dense, Z contains a family degenerating to a torus. In other words, Z^{∗} contains the 0-dimensional cusp in the minimal compactification A_{g}^{∗}. Proposition 2 then implies that Z = A_{g}.

To get around the extra assumptions we placed on A_{0}, we shall show that one can pass from A_{0} to an abelian variety A_{1} isogenous to it, and change the polarization by the isogeny, without changing the question (Q 1). Also we will have to deal with the case that the polarization is not necessarily a product. For this we use an easy algebraic lemma.

The following lemma is well-known. We include a proof for the convenience of the reader.

Lemma 3. Let A be an ordinary absolutely simple abelian variety defined over a finite field 𝔽_{q}, and let K = End_{𝔽_{q}}(A) ⊗_{ℤ} ℚ. Then K is a CM field, [K : ℚ] = 2 dim(A), and K = End_{k}(A) ⊗_{ℤ} ℚ. Let F be the fixed field of the complex conjugation in K, and let π_{q} = Frob_{A/𝔽_{q}} . Then K = ℚ(π_{q}), F = ℚ(π_{q} + q π_{1}^{−1}), and every prime ideal 𝔭 of O_{F} lying over p splits in K.

Proof. For every valuation v of ℚ(π_{q}) lying over p, v(π_{q}) is equal to either v(q) or 0 since A is ordinary. This implies by the the main theorem of [Ta] that the Brauer invariant of the central divisional algebra K/ℚ(π_{q}) at every place over p vanishes. By [Ta] Example a), the field ℚ(π_{q}) is totally imaginary, otherwise A_{0} is a supersingular elliptic curve. Since the central division algebra K/ℚ(π_{q}) splits at all finite places prime to p, we see that K = ℚ(π_{q}). Again by [Ta], [K : ℚ] = 2 dim(A). Example b) of [Ta] shows that K is a totally imaginary quadratic extension of the totally real field F = ℚ(π_{q} + q π_{1}^{−1}) . Since π_{q} and q π_{q}^{−1} are complex conjugate, exactly one of them is a unit for v. Hence every place 𝔭 of F lying over p splits in K. □

Lemma 4. Let K be a CM field and let ∗ be a a positive definite involution of the second kind on M_{n×n}(K). Then there exists a maximal subfield L in M_{n×n}(K) which is stable under ∗. Moreover L is a CM field and [L : ℚ] = [K : ℚ] · n.

Proof. Consider the subspace of 𝔤 of M_{n×n}(K) consisting of elements x ∈ M_{n×n}(K) such that x^{∗} = −x. In other words, 𝔤 is the Lie algebra of the unitary group associated to (M_{n×n}(K), ∗). We know that 𝔤 is a Lie algebra over the totally real subfield F of K, and 𝔤 ⊗_{F} \overline{F} = 𝔤𝔩_{n}(\overline{F}) . The standard representation of M_{n×n}(K) can be regarded as a 2n-dimensional representation of 𝔤 over F. The characteristic polynomial for this representation defines a polynomial f(X, T) of degree 2n in the variable T, with coefficients in the ring S^{•}(𝔤^{∗}) of polynomial functions on 𝔤. It is easy to see that that f(T) is irreducible of degree 2n, but becomes a product of two geometrically irreducible polynomials of degree n in S^{•}(𝔤^{∗})[T] ⊗_{F} K,. By the Hilbert irreducibility theorem, there exists an element x ∈ 𝔤 such that f(x, T) is irreducible over F. This means that the F -subalgebra F [x] of M_{n×n}(K) is a field of degree 2n over F, therefore is equal to K[x]. Take L = F [x] = K[x]. Clearly the subfield L is stable under the involution ∗ and hence is a CM field because ∗ is a positive definite involution. This proves lemma 4. □

Remark. Lemma 4 can certainly be generalized. Group theoretically, it essentially asserts the existence of an anisotropic maximal torus in the special unitary group associated to the simple algebra with involution (M_{n×n}(K), ∗).

Let (A_{0}, λ_{0}) be an ordinary principally polarized abelian variety of dimension g over \overline{𝔽_{p}}. Then A_{0} is isogenous to a product ∏_{i=1}^{m} B_{i}^{n_{i}} , where each B_{i} is a simple ordinary abelian variety over \overline{𝔽_{p}}, and B_{i} is not isogenous to B_{j} if i ≠ j. The last condition implies that

Hom\overline{𝔽_{p}}(B_{i}, B_{j}) = (0) = Hom\overline{𝔽_{p}}(B_{i}, B_{j}^{t}) if i ≠ j ,

where B_{j}^{t} denotes the dual abelian variety of B_{j}. Let K_{i} = End\overline{𝔽_{p}}(B_{i})⊗_{ℤ}ℚ. Lemma 3 tells us that K_{i} is a CM field. Clearly

End\overline{𝔽_{p}}(A_{0}) ⊗_{ℤ} ℚ ≅ ∏_{i=1}^{m} End\overline{𝔽_{p}}(B_{i}^{n_{i}}) ⊗_{ℤ} ℚ ≅ ∏_{i=1}^{m} M_{n_{i}×n_{i}}(K_{i}) .

The pull-back of λ_{0} to ∏_{i=1}^{m} B_{i}^{n_{i}} is necessarily a product ∏_{i=1}^{m} λ_{i}, where λ_{i} is a polarization of B_{i}^{n_{i}} . Therefore the Rosati involution for (A_{0}, λ_{0}) on End\overline{𝔽_{p}}(A_{0})⊗_{ℤ}ℚ can be identified with the product ∏_{i=1}^{m} ∗_{i}, where ∗_{i} is the Rosati involution of (B_{i}^{n_{i}}, λ_{i}) . By Lemma 2, for each i = 1, …, m, there exists a CM field L_{i} ⊆ End\overline{𝔽_{p}}(B_{i}^{n_{i}}) ⊗_{ℤ} ℚ which is stable under the Rosati involution λ_{i}. It is well known that for each i = 1, …, m, there exists an abelian variety A_{i} over \overline{𝔽}_{p} which is isogenous to B_{i}^{n_{i}} and such that End_{\overline{𝔽}_{p}}(A_{i}) contains the ring O_{L_{i}} of all algebraic integers in L_{i}. We conclude that there exist abelian varieties A_{i} over \overline{𝔽}_{p}, i = 1, …, m, an isogeny h : ∏_{i=1}^{m} A_{i} → A_{0} and CM fields L_{i} ⊆ End_{\overline{𝔽}_{p}}(A_{i}) ⊗_{ℤ} ℚ, i = 1, …, m such that

(i) Hom_{\overline{𝔽}_{p}}(A_{i}, A_{j}) = (0) = Hom_{\overline{𝔽}_{p}}(A_{i}, A_{j}^{t}) if i ≠ j; (ii) the pull-back of the polarization λ_{0} on A_{0} to ∏_{i=1}^{m} A_{i} is a product ∏_{i=1}^{m} λ_{i},

where λ_{i} is a polarization of A_{i}, i = 1, …, m; (iii) [L_{i} : ℚ] = 2 dim(A_{i}); (iv) the Rosati involution induced by λ_{i} sends L_{i} to itself and induces the complex

conjugation on L_{i};

(v) O_{L_{i}} ⊆ End_{\overline{𝔽}_{p}}(A_{i}) . Actually property (i) will not be used in what follows.

There exist positive integers d_{1}, …, d_{g}, with d_{1} | ⋯ | d_{g}, such that

Ker (∏_{i=1}^{m} λ_{i}) ≅ (∏_{j=1}^{g} (ℤ/d_{j}ℤ)) × (∏_{j=1}^{g} μ_{d_{j}} )

over \overline{𝔽}_{p}. Let δ = (d_{1}, …, d_{g}) and let K(δ) = (∏_{j=1}^{g} (ℤ/d_{j}ℤ)) × (∏_{j=1}^{g} μ_{d_{j}}). Let A_{g,δ} denote the moduli stack of polarized abelian schemes (A, λ) over a scheme S over Spec(𝔽_{p}) such that Ker(λ) is locally isomorphic to K(δ) in the flat topology. Let A_{g,δ}^{or} be the open algebraic substack of A_{g,δ} whose objects consist of pairs (A, λ) → S in A_{g,δ}(S) such that the all the fibers A_{s} are ordinary abelian varieties. Clearly A_{g,δ}^{or} is equal to A_{g,δ} if d_{1} > 1. By [NO] Thm. 3.1, p. 431, A_{g,δ}^{or} is always dense in A_{g,δ}, see also [Ko] Thm. 7, p. 113 for the case when p does not divide d_{g}. As a consequence of the result on p-adic monodromy in [Ek2] or [FC] V.7.1, A_{g,δ} is irreducible, see [dJ]. Let A_{g}^{or} be the open algebraic substack of ordinary principally polarized abelian varieties in characteristic p. Again by [NO], A_{g}^{or} is dense in A_{g}.

From the construction of (∏_{i=1}^{m} A_{i}, ∏_{i=1}^{m} λ_{i}) we know that Ker(h) ⊆ Ker (∏_{i=1}^{m} λ_{i}) is a maximal totally isotropic subgroup of of Ker (∏_{i=1}^{m} λ_{i}) with respect to the Weil pairing ⟨ , ⟩_{∏_{i} λ_{i}} . Let A_{g,1,δ}^{or}(Ker(h)⊆Ker( ∏_{i} λ_{i})) be the moduli stack of isogenies ϕ : (B, λ_{B}) → (A, λ_{A}) over a scheme S of characteristic p such that (A, λ_{A}) is an ordinary principally polarized abelian variety, λ_{B} is the pull-back ϕ^{∗}(λ_{A}) of λ_{A}, and (Ker(λ_{B}), Ker(ϕ), ⟨ , ⟩_{λ_{B}}) is locally isomorphic to (Ker(∏_{i} λ_{i}), Ker(h), ⟨ , ⟩_{∏_{i} λ_{i}}) in the flat topology. There are natural morphisms

pr_{1} : A_{g,1,δ}^{or}(Ker(h)⊆Ker( ∏_{i} λ_{i})) ⟶ A_{g,δ}^{or} , pr_{2} : A_{g,1,δ}^{or}(Ker(h)⊆Ker( ∏_{i} λ_{i})) ⟶ A_{g}^{or} .

Both morphisms pr_{1}, pr_{2} are finite flat and surjective. Moreover there are prime-to-p Hecke correspondences coming from GSp_{2g}(𝔸_{f}^{(p)}) which act on A_{g,1,δ}^{or}(Ker(h)⊆Ker( ∏_{i} λ_{i})) and A_{g,δ}^{or} as algebraic correspondences, such that the morphisms pr_{1}, pr_{2} are compatible with the prime-to-p Hecke correspondences. Consequently, if ϕ : (B, λ_{B}) → (A, λ_{A}) is a geometric point of A_{g,1,δ}^{or}(Ker(h)⊆Ker( ∏_{i} λ_{i})) , then the prime-to-p Hecke orbit (resp. ℓ-power Hecke orbit) of the point [(B, λ)] is Zariski dense in A_{g,δ}^{or} if and only if the prime-to-p Hecke orbit (resp. ℓ-power Hecke orbit) of the point [(A, λ_{A})] is Zariski dense in A_{g}^{or}. Therefore the prime-to-p Hecke orbit (resp. ℓpower Hecke orbit) of the point [(B, λ)] is Zariski dense in A_{g,δ} if and only if the prime-to-p Hecke orbit (resp. ℓ-power Hecke orbit) of the point [(A, λ_{A})] is Zariski dense in A_{g}. The same consideration also leads to

Proposition 2′′. Let k be an algebraically closed field of characteristic p > 0. Let ℓ be a prime number different from p. Let d_{1}, …, d_{g} be positive integers such that d_{|} ⋯ |d_{g}, and let δ = (d_{1}, …, d_{g}) . Denote by A_{g,δ}^{or} the moduli stack of polarized ordinary abelian varieties over k as above. Suppose that Z ⊆ A_{g,δ}^{or} is a closed subscheme of A_{g,δ}^{or} which is stable under all ℓ-power Hecke correspondences. Assume furthermore that there exists a nonconstant morphism C → Z from a smooth open curve C over k to Z, such that the pull-back of the universal abelian scheme over C degenerates to a torus at some boundary point of C. Then Z = A_{g,δ}.

Proof. The hypothesis implies that the Zariski closure W^{∗} of W = pr_{2} (pr_{1}^{−1}(Z)) in A_{g}^{∗} contains the standard 0-dimensional cusp and is stable under all prime-to-p Hecke correspondences. Therefore W = A_{g}^{or} by Proposition 2. This implies that Z = A_{g,δ}. □

The above discussion clearly applies to the isogeny h : (∏_{i=1}^{m} A_{i}, ∏_{i+1}^{m} λ_{i}) → (A_{0}, λ_{0}). From Proposition 2′′, we see that the Zariski closure Z of the prime-to-p Hecke orbit of the point [(A_{0}, λ_{0})] in A_{g} is equal to A_{g} if and only if the Zariski closure Z′ of the prime-to-p orbit of the point [(∏_{i=1}^{m} A_{i}, ∏_{i+1}^{m} λ_{i})] in A_{g,δ} contains a 1-parameter family of abelian varieties degenerating to a torus. This will clearly be the case if for each i, the Zariski closure of the prime-to-p Hecke orbit of the point [(A_{i}, λ_{i})] in A_{g_{i},δ_{i}} contains a one-parameter family of abelian varieties degenerating to a torus. Here g_{i} = dim(A_{i}), and K(δ_{i}) ≅ Ker(λ_{i}) geometrically. So far we have reduced our original question (Q 1) to a similar question on the Zariski closure of the prime-to-p Hecke orbit of an ordinary polarized abelian variety (A_{1}, λ_{1}) in A_{g,δ_{1}}(\overline{𝔽}_{p}), where (A_{1}, λ_{1}) has the property that there exists a CM field L with [L : ℚ] = 2 dim(A_{1}) and an embedding O_{L} ⊆ End_{\overline{𝔽}_{p}}(A_{1}) such that the Rosati involution on End_{\overline{𝔽}_{p}}(A_{1})⊗_{ℤ} ℚ induced by λ_{1} sends L to L and induces the complex conjugation on L.

Suppose (A_{1}, λ_{1}) is an ordinary polarized abelian variety with the above properties. Let F be the totally real subfield in L, namely the subfield of L consisting of all totally real elements in L. Let O_{F} be the ring of algebraic integers in F, g = [F : ℚ]. Denote by M_{O_{F}}, or for short M_{F}, the moduli stack of abelian schemes of dimension g with multiplication by O_{F} as in [DP] 2.1. In other words M_{F} is the stack of groupoids whose objects are pairs (A → S, ι), where A → S is an abelian scheme of relative dimension g and ι : O_{F} → End(A/S) gives A the structure of an O_{F} -module. Given (A → S, ι) as above, let

P = P(A, ι) = \underline{Hom}_{O_{F}} (A, A^{t})^{sym}

be the étale sheaf over S of O_{F} -linear quasi-polarizations of (A, ι) . Let P_{+} ⊂ P be the étale sheaf over S of O_{F} -linear polarizations of (A, ι) . We know that P is an étale sheaf over S of projective O_{F} -modules of rank 1, and P_{+} defines a notion of positivity on P. In other words, for every real embedding σ of F we have an orientation of P ⊗_{O_{F} ,σ} ℝ, and P_{+} is the subsheaf of totally positive elements. Denote by O_{F,+}^{×} the group of totally positive units in O_{F}. Then (P, P_{+}) is an O_{F,+}^{×}-torsor, which has to be constant if S is normal and connected, c.f. [Ra] 1.18. By [DP] Théorème 2.2 M_{F} is geometrically normal over Spec(ℤ), hence (P, P_{+}) is constant on each connected component of M_{F}.

We can also specify the sheaf of polarizations and arrive at a variant of the above definition. Assume that Λ is a projective O_{F} -module of rank 1 with a notion of positivity; that is for every real embedding σ of F we are given an orientation of Λ ⊗_{O_{F} ,σ} ℝ. We can define a moduli stack M_{O_{F}}^{(Λ, Λ_{+})} = M_{F}^{Λ} whose objects consists of triples (A → S, ι, h) where (A → S, ι) is an abelian schemes of relative dimension g with multiplication by O_{F} as before, and

h : A ⊗_{O_{F}} Λ ⟶^{∼} A^{t}

is an isomorphism, which sends λ ∈ Λ to ϕ_{λ} ∈ P(A, ι) and P_{+}(A, ι) to Λ_{+} . The algebraic stack M_{F} is isomorphic to a disjoint union ∐_{i} M_{F}^{Λ_{i}} , where Λ_{i} runs through a system of representatives of the strict ideal class group of F. Notice that we have used the definition of the Hibert-Blumenthal moduli spaces given in [DP] 2.1, which modifies the definition in [Ra] 1.1.

There are natural Hecke correspondences on M_{F} coming from GL_{2}(𝔸_{f,F}) , the group of finite adelic points of GL(2, F). To distinguish the Hecke orbits on the Hilbert-Blumenthal moduli spaces from the Hecke orbits on the Siegel moduli spaces, we shall use the term ‘F -Hecke orbits’ for Hecke orbits on M_{F}, M_{F}^{Λ} or their variants. For instance the prime-to-p F -Hecke orbits are the orbits of prime-to-p Hecke correspondences which come from GL_{2}(𝔸_{f,F}^{(p)} ) ; the ℓ-power F -Hecke orbits are the orbits of ℓ-power Hecke correspondences which come from GL_{2}(ℚ_{ℓ}) . Here

𝔸_{f,F}^{(p)} = 𝔸_{ℚ,F}^{(p)} ⊗_{ℚ} F = (∏_{ℓ≠p}′ ℚ_{ℓ}) ⊗_{ℚ} F

is the ring of finite prime-to-p adeles of F. If [(A, ι_{A})] is a point of M_{F} over an algebraically closed field of characteristic p > 0, the prime-to-p F -Hecke orbit can be described geometrically as follows. It consists of all points of M_{F} (k) of the form [(B, ι_{B})] such that (B, ι_{B}) is isogenous to (A, ι_{A}) via an O_{F} -linear isogeny whose kernel is killed by some prime-to-p integer. For M_{F}^{Λ} the F -Hecke operators coming from GL_{2}(𝔸_{f,F}) do not preserve M_{F}^{Λ} in general. Consider a point [(A, ι_{A}, h_{A})] in M_{F}^{Λ}(k) , where k is an algebraically closed field of characteristic p > 0, and h_{A} is thought of as an O_{F} -linear isomorphism from (Λ, Λ_{+}) to (P(A, ι_{A}), P(A, ι_{A})_{+}). The intersection of M_{F}^{Λ}(k) with the GL_{2}(𝔸_{f,F})-Hecke orbit of [(A, ι_{A}, h_{A})] can be described as follows. It consists of all points [(B, ι_{B}, h_{B})] in M_{F}^{Λ}(k) such that there exists an O_{F} -linear isogeny ϕ : B → A and a totally positive element c ∈ O_{F} such that Ker(ϕ) is killed by some prime-to-p integer, and

tϕ ∘ h_{A}(λ) ∘ ϕ = c · h_{B}(λ) for every λ ∈ Λ .

In terms of group theory, an element γ of GL_{2}(𝔸_{f,F}^{(p)} ) changes the isomorphism class of the polarization sheaf via the translation by an element of the strict ideal class group; this element in the strict ideal class group comes from the determinant of γ.

From now on we shall restrict ourselves to schemes over 𝔽_{p}, therefore M_{F} and M_{F}^{Λ} stand for the Hilbert moduli stacks over 𝔽_{p}. Let M_{F}^{or} and M_{F}^{Λ,or} be the ordinary locus of M_{F} and M_{F}^{Λ} respectively. It is known that M_{F}^{or} (resp. M_{F}^{Λ,or}) is open and dense in M_{F} (resp. M_{F}^{Λ}).

Now we return to the previous situation: A_{1} is an ordinary abelian variety over \overline{𝔽}_{p}, λ_{1} is a polarization of A_{1}, L is a CM field with [L : ℚ] = 2 dim(A), O_{L} operates on A_{1} via an embedding ι̃_{1} : O_{F} → End(A_{1}), and the Rosati involution ∗_{λ_{1}} sends L into itself and induces the complex conjugation on L. Let ι_{1} : O_{F} → End(A_{1}) be the restriction of ι̃_{1} to O_{F}. Then (A_{1}, ι_{1}) is a point of M_{F} (\overline{𝔽}_{p}). The polarization λ_{1} gives a section of P(A_{1}, ι_{1})_{+} over Spec(\overline{𝔽}_{p}). The polarization λ_{1} on A_{1} has a type δ, characterized by K(δ) ≅ Ker(λ_{1}). If we write δ = (d_{1}, …, d_{g}), d_{1} | ⋯ | d_{g}, then P(A_{1}, ι_{1})/O_{F} · λ_{1} is isomorphic to ℤ/d_{1}ℤ ⊕ ⋯ ⊕ ℤ/d_{g}ℤ as abelian groups. Let (Λ, Λ_{+}) = (P(A_{1}, ι_{1}), P(A_{1}, ι_{1})_{+}) , so that λ_{1} is an element of Λ_{+} ⊂ Λ.

We define a morphism

j : M_{F}^{Λ,or} ⟶ A_{g,δ}^{or}

which sends a triple (A → S, ι, h) in M_{F}^{Λ,or}(S) to the polarized ordinary abelian variety (A → S, h(λ_{1})) in A_{g,δ}^{or}(S) , where h(λ_{1}) is the O_{F} -linear polarization given by the element λ_{1} in Λ. On A_{g,δ}^{or} there are prime-to-p Hecke correspon-

dences coming from the group GSp(𝔸_{f,ℚ}^{(p)}) . We shall explain the compatibility of the morphism j with Hecke operators. First we need some notation. The first homology group of A_{1} with coefficients in 𝔸_{f,ℚ}^{(p)} ,

V_{f}^{(p)}(A_{1}) = H^{1}(A_{1}, 𝔸_{f,ℚ}^{(p)}) := ∏_{ℓ≠p}′ H^{1}(A_{1}, ℚ_{ℓ}) ,

is a free 𝔸_{f,ℚ}^{(p)}-module of rank 2 dim(A_{1}) = [L : ℚ] . The polarization h(λ_{1}) of A_{1} gives a nondegenerate symplectic pairing

⟨ , ⟩_{λ_{1}} : V_{f}^{(p)}(A_{1}) × V_{f}^{(p)}(A_{1}) ⟶ 𝔸_{f,ℚ}^{(p)}(1) .

This defines a group of symplectic similitudes, which we denote by GSp(V_{f}^{(p)}(A_{1}), λ_{1}) , or GSp(V_{f}^{(p)}(A_{1})) for short. It is isomorphic to GSp_{2g}(𝔸_{f,ℚ}^{(p)}). Now we turn to the other side for M_{F}^{Λ}. The group GL_{𝔸_{f,ℚ}^{(p)}} (V_{f}^{(p)}(A_{1})) is isomorphic to GL_{2}(𝔸_{f,ℚ}^{(p)}). But GL_{𝔸_{f,ℚ}^{(p)}} (V_{f}^{(p)}(A_{1})) does not embed into GSp(V_{f}^{(p)}(A_{1})) in a natural way. We have to cut down the size of the center of GL_{𝔸_{f,ℚ}^{(p)}} (V_{f}^{(p)}(A_{1})) to make that happen. The nondegenerate symplectic pairing ⟨ , ⟩_{λ_{1}} factorizes through an 𝔸_{f,F}^{(p)} - linear nondegenerate alternating pairing

⟨ , ⟩_{F} : V_{f}^{(p)}(A_{1}) × V_{f}^{(p)}(A_{1}) ⟶ 𝔸_{f,𝔽}^{(p)}(1)

such that ⟨ , ⟩_{λ_{1}} = tr_{F/ℚ} ∘⟨ , ⟩_{F} . Since ⋀_{𝔸_{f,𝔽}^{(p)}}^{2} (V_{f}^{(p)}(A_{1})) is a free 𝔸_{f,𝔽}^{(p)}-module of rank 1, the involution ∗ = ∗_{F} on End_{𝔸_{f,𝔽}^{(p)}} (V_{f}^{(p)}(A_{1})) induced by ⟨ , ⟩_{F} is the standard involution. Especially

α · α^{∗} = α^{∗} · α = det(α) · Id , ∀ α ∈ End_{𝔸_{f,𝔽}^{(p)}} (V_{f}^{(p)}(A_{1})) .

Let GL_{𝔸_{f,F^{(p)}}}^{red} (V_{f}^{(p)}(A_{1})) be the subgroup of GL_{𝔸_{f,F^{(p)}}} (V_{f}^{(p)}(A_{1})) consisting of all elements α ∈ GL_{𝔸_{f,F^{(p)}}} (V_{f}^{(p)}(A_{1})) such that det(α) ∈ 𝔸_{f,ℚ}^{(p) ×}. Clearly the center of the group GL_{𝔸_{f,F^{(p)}}}^{red} (V_{f}^{(p)}(A_{1})) is 𝔸_{f,ℚ}^{(p) ×}, and the derived group of GL_{𝔸_{f,F^{(p)}}}^{red} (V_{f}^{(p)}(A_{1})) is SL_{𝔸_{f,F^{(p)}}} (V_{f}^{(p)}(A_{1})). The action of GL_{𝔸_{f,F^{(p)}}}^{red} (V_{f}^{(p)}(A_{1})) on V_{f}^{(p)}(A_{1}) identifies the group GL_{𝔸_{f,F^{(p)}}}^{red} (V_{f}^{(p)}(A_{1})) as a subgroup of GSp (V_{f}^{(p)}(A_{1})).

We go back to the morphism

j : M_{F}^{Λ,or} ⟶ A_{g,δ}^{or} .

Clearly j extends naturally to a morphism j^{(p)} from the prime-to-p tower of M_{F}^{Λ,or} to the prime-to-p tower of A_{g,δ}^{(or)} . The group GL_{𝔸_{f,F^{(p)}}}^{red} (V_{f}^{(p)}(A_{1})) operates on the prime-to-p tower of M_{F}^{Λ,or}, while GSp (V_{f}^{(p)}(A_{1})) operates on the prime-to-p tower of A_{g,δ}^{or}. Moreover j^{(p)} is equivariant with respect to the embedding of GL_{𝔸_{f,F^{(p)}}}^{red} (V_{f}^{(p)}(A_{1})) in GSp (V_{f}^{(p)}(A_{1})). These statements are not hard to check, and we omit their verification since we do not need it. For us the important consequence is that the image of the GL_{𝔸_{f,F^{(p)}}}^{red} (V_{f}^{(p)}(A_{1}))-Hecke orbit of a point [(A_{1}, ι_{1}, h_{1})] in M_{F}^{Λ,or}(\overline{𝔽_{p}}) is contained in the prime-to-p Hecke orbit of the point [(A_{1}, λ_{1})] in A_{g,δ}^{(or)}(\overline{𝔽_{p}}) . For an algebraically closed field k of characteristic p > 0, the GL_{𝔸_{f,F^{(p)}}}^{red} (V_{f}^{(p)}(A_{1}))-Hecke orbit of the point [(A_{1}, ι_{1}, h_{1})] in M_{F}^{Λ,or}(\overline{𝔽_{p}})(k) is explicitly given below, and will be referred to as the reduced prime-to-p F -Hecke orbit of the point [(A_{1}, ι_{1}, h_{1})] in M_{F}^{Λ,or}(\overline{𝔽_{p}})(k) . It consists of all points [B, ι_{B}, h_{B})] in M_{F}^{Λ}(\overline{𝔽_{p}}) such that there exists an O_{F} -linear isogeny ϕ : B → A, a totally positive unit u ∈ O_{F}^{×} and a positive integer n ∈ ℤ_{(p)} such that

tϕ ∘ h_{A}(λ) ∘ ϕ = n · u · h_{B}(λ) , ∀ λ ∈ Λ .

It is clear from this description that j sends the reduced prime-to-p F -Hecke orbit of the point [(A_{1}, ι_{1}, h_{1})] in M_{F}^{Λ,or}(\overline{𝔽_{p}}) into the prime-to-p Hecke orbit of the point [(A_{1}, λ_{1})] in A_{g,δ}^{or}(\overline{𝔽_{p}}) . Of course for a positive integer D with (D, p) = 1 , there is a similar notion of ‘reduced D-power F -Hecke orbit’ in M_{F}^{Λ,or}(k) , and j sends a reduced D-power F -Hecke orbit in M_{F}^{Λ,or}(k) into a D-power Hecke orbit of A_{g,δ}^{or}(k) .

Proposition 3. Let k be an algebraically closed field of characteristic p > 0. Let D be a positive integer with (D, p) = 1. Assume that the reduced prime-to-p (resp. reduced D-power) F -Hecke orbit of every ordinary point of M_{F}^{Λ}(\overline{𝔽_{p}}) is Zariski dense in M_{F}^{Λ}, for any totally real number field F and any projective rank-one O_{F} - module Λ with a notion of positivity Λ_{+} . Then the prime-to-p (resp. D-power) Hecke orbit of every ordinary point in A_{g,δ}(k) is dense in A_{g,δ}, for any g > 0 and any type δ.

Proof. Recall that at the boundary of M_{F}^{Λ} , the universal abelian scheme with multiplication by O_{F} degenerates to a torus. Therefore if the reduced prime-to-p (resp. D-power) F -Hecke orbit of (A_{1}, ι_{1}) is Zariski dense in M_{F}^{Λ} , then the Zariski closure Z of the prime-to-p (resp. D-power) Hecke orbit of (A_{1}, λ_{1}) contains a one-parameter family which degenerates to a torus. Proposition 2′′ concludes the proof. □

In the rest of this section we collect a few facts about Hilbert-Blumenthal abelian varieties. They are all well known. Proofs are supplied for the convenience of the reader.

Lemma 5. Let F be a totally real number field. Let (A_{1}, ι_{1}) be an ordinary abelian variety of dimension [F : ℚ] over a perfect field k of characteristic p > 0 with multiplication ι_{1} : O_{F} → End_{k}(A_{1}) by O_{F}. Then its Lie algebra Lie(A_{1}) is a free module over O_{F} ⊗_{ℤ} k of rank 1.

Proof. The p-divisible group A_{1}[p^{∞}] associated to A_{1} is the direct sum of its toric part A_{1}[p^{∞}]_{toric} and its étale part A_{1}[p^{∞}]_{ét}. Therefore its contravariant Dieudonné module 𝔻 (A_{1}[p^{∞}]) is the direct sum 𝔻 (A_{1}[p^{∞}]_{toric}) ⊕ 𝔻 (A_{1}[p^{∞}]_{ét}) of the Diedonné modules of its toric and étale part. By [Ra] 1.3, 𝔻 (A_{1}[p^{∞}]) is a free O_{F} ⊗_{ℤ}W (k)-module of rank 2. Notice that [Ra] 1.3 is formulated generally and does not depend on the freeness condition (*) in Def. 1.1, loc. cit. Hence 𝔻 (A_{1}[p^{∞}]_{toric}) is a projective module over O_{F} ⊗_{ℤ}W (k). Since 𝔻 (A_{1}[p^{∞}]_{toric})⊗_{W (k)}k ≅ Lie(A_{1})^{∨}, Lie(A_{1})^{∨} is a projective module over O_{F} ⊗_{ℤ} k. This implies that Lie(A_{1}) is a projective module over O_{F} ⊗_{ℤ} k. On the other hand by [DP] 2.7, we know that Lie(A_{1}) and O_{F} ⊗_{ℤ} k have the same class in the Grothendieck group of O_{F} ⊗_{ℤ} k-modules of finite type. Taking into account that O_{F} ⊗_{ℤ} k is an artinian k-algebra, this implies that Lie(A_{1}) is isomorphic to O_{F} ⊗_{ℤ} k as an O_{F} ⊗_{ℤ} k-module. □

Lemma 6. Let F be a totally real number field. Let (A, ι) be an abelian variety of dimension g = [F : ℚ] with multiplication by O_{F} over an algebraically closed field k. Then A is isogenous to B^{n} for some simple abelian variety B over k. Let D = End_{k}(B) ⊗_{ℤ} ℚ, so End_{k}(A) ⊗_{ℤ} ℚ ≅ M_{n×n}(D) . There are four possibilities i. (Type I) The algebra D is a totally real number field K_{0}, and [K_{0} : ℚ] =

dim(B). F contains K_{0}, and [F : K_{0}] = n. The centralizer of F in End_{k}(A) ⊗_{ℤ} ℚ is equal to F itself.

ii. (Type II) The algebra D is a totally indefinite quaternion division algebra over a totally real number field K_{0}, and dim(B) = 2 [K_{0} : ℚ]. The field F contains K_{0}, and [F : K_{0}] = 2n. The centralizer of F in End_{k}(A) ⊗_{ℤ} ℚ is equal to F itself.

iii. (Type III) The algebra D is a totally definite quaternion division algebra over a totally real number field K_{0}. There are 2 cases:

(a) The field k has characteristic p, K_{0} = ℚ and dim(B) = 1. The algebra D is the quaternion division algebra over ℚ ramified only at p and ∞. The algebra B is a supersingular elliptic curve over k. The centralizer of F in End_{k}(A) ⊗_{ℤ} ℚ is the quaternion division algebra over F which is ramified at all infinite places of F and all places v of F above p such that [F_{v} : ℚ_{p}] is odd, and is unramified at all other finite places.

(b) The simple abelian variety B is not an elliptic curve; dim(B) = 2 [K_{0} : ℚ]. The field F contains K_{0}, and [F : K_{0}] = 2n. The centralizer of F in End_{k}(A) ⊗_{ℤ} ℚ is equal to F itself.

iv. (Type IV) The algebra D is a central division algebra over a totally imaginary quadratic extension K of a totally real number field K_{0}. Write dim_{K}(D) = d^{2}. Then F ⊇ K_{0}, and [F : K_{0}] = n d. The centralizer of F in End_{k}(A) ⊗_{ℤ} ℚ is equal to F · K, the subalgebra in End_{k}(A) ⊗_{ℤ} ℚ generated by F and K. It is isomorphic to F ⊗_{K_{0}} K, a totally imaginary quadratic extension of F. If char(k) = 0, then d = 1 and D = K.

The abelian variety A does not have sufficiently many complex multiplication (smCM) in the following cases: (Type I), (Type II), (Type III (b) ) , and has smCM in the other cases. If k = \overline{𝔽}_{p} and A is ordinary , then D is a CM-field, an example of the (Type IV) case. Here to say that A has smCM means that End^{0}(A) contains a commutative semisimple subalgebra of dimension 2 dim(A) over ℚ.

Proof. The abelian variety A is isogenous to a product ∏_{i=1}^{m} B_{i}^{n_{i}} , where B_{i} is not isogenous to B_{j} if i ≠ j. Let B = B_{1} , n = n_{1} , D = End_{k}(B_{1}) ⊗_{ℤ} ℚ. The natural map of F → M_{n×n}(D) is an embedding. The division algebra D has one of the four possibilities as in [Mu], p. 202. See also [Sh] §2 for the characteristic 0 case. i. (Type I) The algebra D is a totally real algebraic number field. The table in [Mu] p. 202 gives [K_{0} : ℚ] | dim(B), hence n [K_{0} : ℚ] ≤ n dim(B) ≤ [F : ℚ]. Since F is a commutative semisimple subalgebra of M_{n×n}(D), [F : ℚ] ≤ n [K_{0} : ℚ]. Therefore n [K_{0} : ℚ] = n dim(B) = dim(A) = [F : ℚ] = dim(A) and m = 1. This implies that F is a maximal commutative semisimple subalgebra of M_{n×n}(K_{0}), therefore F contains K_{0}. ii. (Type II) The algebra D is a totally indefinite quaternion division algebra over a totally real field K_{0}. In this case [Mu] p. 202 gives 2 [K_{0} : ℚ] | dim(B). Hence 2n [K_{0} : ℚ] ≤ n dim(B) ≤ dim(A) = [F : ℚ]. On the other hand [F : Q] ≤ 2n [K_{0} : ℚ] since F is a commutative semisimple subalgebra of M_{n×n}(D). So [F : Q] = 2n [K_{0} : ℚ], 2 [K_{0} : ℚ] = dim(B), and m = 1. This implies that F is a maximal commutative semisimple subalgebra of M_{n×n}(D), therefore F contains K_{0}. iii. (Type III) The algebra D is a totally definite quaternion division algebra over a totally real number field K_{0}. The table in [Mu], p. 202 gives [K_{0} : ℚ] | dim(B), and 2 [K_{0} : ℚ] | dim(B) if char.(k) = 0. First assume that [K_{0} : ℚ] = dim(B). In this case the field k has to have positive characteristic. The abelian variety B has smCM. Grothendieck has shown that in positive characteristic, an abelian variety with smCM is isogenous to an abelian variety defined over a finite field, see [Oort]. Therefore we may assume that B is defined over a finite field 𝔽_{q}. Moreover we may assume that all elements of End_{\overline{𝔽}_{q}}(B) are defined over 𝔽_{q}. The Frobenius element π_{B/𝔽q} for A_{1}/𝔽_{q} belongs to K_{0} because it commutes with all elements of End_{\overline{𝔽}_{q}}(B). Since K_{0} has real places, we conclude that B is a supersingular elliptic curve; see [Ta], §1 example a). The rest of the statement (a) follow from standard results on central simple algebras.

Next assume that dim(B) ≥ 2 [K_{0} : ℚ]. Then as in the (Type II) case we have 2n [K_{0} : ℚ] ≤ n dim(B) ≤ dim(A) = [F : ℚ] and [F : Q] ≤ 2n [K_{0} : ℚ] because F is a commutative semisimple subalgebra of M_{n×n}(D). Therefore [F : Q] = 2n [K_{0} : ℚ], 2 [K_{0} : ℚ] = dim(B), m = 1 and F contains K_{0}. iv. (Type IV) The algebra D is a central division algebra over a totally imaginary quadratic extension K of a totally real number field K_{0}. Write dim_{K}(D) = d^{2}. The table in [Mu], p. 202 says that [K_{0} : ℚ] d | dim(B), and [K_{0} : ℚ] d^{2} | dim(B) if char.(k) = 0. So n d [K_{0} : ℚ] ≤ n dim(B) ≤ dim(A) = [F : ℚ]. On the other hand [F : ℚ] ≤ n d [K_{0} : ℚ] since F · K is a commutative semisimple subalgebra of M_{n×n}(D). Therefore [F : ℚ] = n d [K_{0} : ℚ], m = 1, d [K_{0} : ℚ] = dim(B), F ⊇ K_{0} and [F : K_{0}] = n d. If char.(k) = 0, we have [K_{0} : ℚ] d^{2} | dim(B), therefore d = 1 and D = K. To compute the centralizer of F in M_{n×n}(D), we use descent from K to K_{0}: M_{n×n}(D) ⊗_{K_{0}} K is isomorphic to (M_{n×n}(D) ⊗_{K_{0}} K) × (M_{n×n}(D) ⊗_{K_{0}} K) . Via this isomorphism F ⊗_{K_{0}} K becomes a subalgebra of (M_{n×n}(D) ⊗_{K_{0}} K) × (M_{n×n}(D) ⊗_{K_{0}} K) . Its image in either factor is equal to F · K, a maximal commutative semisimple subalgebra of M_{n×n}(D) . Therefore the centralizer of F ⊗_{K_{0}} K is contained in (F · K) × (F · K) = (F × K) ⊗_{K_{0}} K. Consequently the centralizer of F in M_{n×n}(D) is contained in F · K. The other inclusion is obvious.

When k = \overline{𝔽}_{p} and A is ordinary, we know that D is a CM-field by Lemma 3, therefore belongs to (Type IV). □

The following lemma is an analogue of Proposition 1 for Hilbert-Blumenthal abelian varieties.

Lemma 7. Let F be a totally real algebraic number field field, O_{F} be the ring of algebraic integers in F. Let k be an algebraically closed field of characteristic p > 0. Assume that A_{1} is an abelian variety over k, with a structure of O_{F} -module ι_{1} : O → End(A). If the reduced ℓ-power F -Hecke orbit of (A_{1}, ι_{1}) in M_{F} is finite for a prime number ℓ different from p, then A_{1} is supersingular. Conversely, if A_{1} is supersingular, then the prime-to-p F -Hecke orbit of (A_{1}, ι_{1}) in M_{F} is finite.

Proof. Since the proof of this lemma is very similar to the proof of Proposition 1, we shall only give a sketch and leave the complete proof to the reader as an exercise. First if A_{1} is supersingular, then End(A_{1}, ι_{1}) ⊗_{ℤ} ℚ is the centralizer of F in End(A_{1}) ⊗_{ℤ} ℚ, hence is a quaternion algebra over F. That is it is a central simple algebra over F of relative dimension 4. The expression of the prime-to-p F - Hecke orbit of (A_{1}, ι_{1}) in M_{F} as a double coset shows that it is finite. Conversely, assume that the reduced ℓ-power F -Hecke orbit of (A_{1}, ι_{1}) in M_{F} is finite for a prime number ℓ ≠ p. Then we conclude as in the proof of Proposition 1 that the F ⊗_{ℚ} ℚ_{ℓ}-linear span of

(End(A_{1}, ι_{1}) ⊗_{ℤ} ℚ)^{×} ∩ GL_{F ⊗_{ℚ}ℚ_{ℓ}}^{red}(H_{1}(A_{1}), ℚ_{ℓ})

is End_{F ⊗_{ℚ}ℚ_{ℓ}}(H_{1}(A_{1}), ℚ_{ℓ}) . Hence End(A_{1}, ι_{1}) ⊗_{ℤ} ℚ is a quaternion algebra over F. An inspection of the cases in Lemma 6 finishes the proof. Alternatively we can finish the proof directly as follows: The abelian variety A_{1} has smCM, therefore is isogenous to an abelian variety A_{2} defined over a finite field 𝔽_{q}, and we may assume that all elements of End_{\overline{𝔽}_{q}}(A_{2}) are defined over 𝔽_{q}. The Frobenius element π_{A_{2}/𝔽q} for A_{1}/𝔽_{q} belongs to F because it commutes with all elements of End_{\overline{𝔽}_{q}}(A_{2}). Since F has real places, we conclude that A_{2} is supersingular, so is A_{1}. See [Ta], §1 Example a). □

Lemma 8. Suppose that F be a totally real number field, (A, ι) is an ordinary abelian variety of dimension g = [F : ℚ] over a perfect field k of characteristic p > 0 with multiplication ι : O_{F} → End_{k}(A) by O_{F}. Let T_{p} (A[p^{∞}]_{ét}) be the p-adic Tate module for the étale quotient of A[p^{∞}] , V_{p} (A[p^{∞}]_{ét}) = T_{p} (A[p^{∞}]_{ét}) ⊗_{ℤ_{p}} ℚ_{p}. Similarly for the dual A^{t} of A. Then

(1) The p-adic Tate modules T_{p} (A[p^{∞}]_{ét}) and T_{p} (A^{t}[p^{∞}]_{ét}) are both free O_{F} ⊗_{ℤ}ℤ_{p}- modules of rank 1.

(2) If (A, ι) is defined over a finite field, then (T_{p} (A[p^{∞}]_{ét}) ⊕ T_{p} (A^{t}[p^{∞}]_{ét})) ⊗_{ℤ_{p}} ℚ_{p} is a free module over L ⊗_{ℚ} ℚ_{p} of rank 1, where L = End_{k}(A, ι) ⊗_{ℤ} ℚ is the endomorphism algebra of the abelian variety (A, ι) up to isogeny with multiplication by F. The field L is a totally imaginary quadratic extension of F which is split at every place 𝔭 of F above p. If moreover O_{L} = End_{k}(A, ι) , then T_{p} (A[p^{∞}]_{ét}) ⊕ T_{p} (A^{t}[p^{∞}]_{ét}) is a free O_{L} ⊗_{ℤ} ℤ_{p}-module of rank 1.

Proof. (1) Since T_{p} (A[p^{∞}]_{ét}) is torsion-free, it is a projective module over O_{F} ⊗_{ℤ} ℤ_{p}. We know that Lie(A^{t})^{∨} ≅ T_{p} (A[p^{∞}]_{ét})⊗_{ℤ_{p}} k as O_{F} ⊗_{ℤ} k-modules, and Lie(A^{t}) is a free O_{F} ⊗_{ℤ} k-module of rank 1 by Lemma 5. Hence T_{p} (A[p^{∞}]_{ét}) is a free O_{F} ⊗_{ℤ} ℤ_{p}-module of rank 1. Similarly for A^{t}.

(2) If O_{L} = End_{k}(A, ι) , then T_{p} (A[p^{∞}]_{ét}) ⊕ T_{p} (A^{t}[p^{∞}]_{ét}) is a projective module over O_{L} ⊗_{ℤ} ℤ_{p} because it is torsion-free. So the second statement in (2) follows from the first one. We know from Lemma 6 that we are in the Type IV case of Lemma 6. Therefore L is a totally imaginary quadratic extension of F. In the notation of Lemma 6 iv, we have L = F · K, and the quadratic extension K/K_{0} is split at every place of K_{0} above p by Lemma 3, therefore the extension L/F is split at every place of F above p. Clearly (T_{p} (A[p^{∞}]_{ét}) ⊕ T_{p} (A^{t}[p^{∞}]_{ét}^{∨})) ⊗_{ℤ_{p}} ℚ_{p} is a projective module over L ⊗_{ℚ} ℚ_{p}. Moreover we know that the trace of the natural linear action of any element of L on (T_{p} (A[p^{∞}]_{ét}) ⊕ T_{p} (A^{t}[p^{∞}]_{ét}^{∨})) ⊗_{ℤ_{p}} ℚ_{p} is in ℚ, because as an L ⊗ W (k)-module (T_{p} (A[p^{∞}]_{ét}) ⊕ T_{p} (A^{t}[p^{∞}]_{ét}^{∨})) ⊗_{ℤ_{p}} 𝕎(k) is naturally isomorphic to the W (k)-dual of the first crystalline cohomology group of A/W (k). So (T_{p} (A[p^{∞}]_{ét}) ⊕ T_{p} (A^{t}[p^{∞}]_{ét}^{∨})) ⊗_{ℤ_{p}} ℚ_{p} is a free module of rank 1 over L ⊗_{ℚ} ℚ_{p}. Since over a discrete valuation ring every free module of finite rank is non-canonically isomorphic to its dual, it follows that (T_{p} (A[p^{∞}]_{ét}) ⊕ T_{p} (A^{t}[p^{∞}]_{ét})) ⊗_{ℤ_{p}} ℚ_{p} is a free module over L ⊗_{ℚ} ℚ_{p} of rank 1. □

4. Calculation at smooth ordinary points over finite fields

We keep the notation in the previous section: F is a totally real algebraic number field, O_{F} denotes the ring of algebraic integers in F. Let k be an algebraically closed field of characteristic p > 0. We have the Hilbert moduli space M_{F} and its variants M_{F}^{Λ}.

Suppose that Z̃ is the Zariski closure of the reduced prime-to-p Hecke orbit of an ordinary point in M_{F}^{Λ}(k) . Then the intersection of the open subset of smooth points of Z with M_{F}^{Λ,or} is an open dense subscheme Z̃_{sm}^{or} of Z̃. Moreover Z̃_{sm}^{or} is stable under the reduced prime-to-p F -Hecke correspondences, and contains a dense subset of points defined over finite fields. Therefore for any point x̃ ∈ Z̃_{sm}^{or} defined over \overline{𝔽}_{p}, the formal completion Z̃∧ ; x̃ of Z̃ at x̃ is stable under the action of the stabilizer subgroup of x̃ in the reduced prime-to-p F -Hecke correspondences. In this section we shall decipher this information by an explicit calculation using the Serre-Tate coordinates.

We first recall the Serre-Tate coordinates. A good reference is [Ka2]. Let A_{1} be an ordinary abelian variety over k. Denote by T_{p}(A_{1})(k) the ‘physical’ p-adic Tate module T_{p} (A_{1} [p^{∞}]_{ét}) (k) . Let Def_{A_{1}} be the deformation functor which sends a pair (R, ϵ) consisting of an artinian local ring R and an isomorphism ϵ : κ_{R} = R/𝔪_{R} ⟶ k to

Def_{A_{1}}(R, ϵ) = \{(A, α) | \begin{matrix}A is an abelian scheme over R; and \\ α : A ⊗_{R,ϵ} k ⟶^{∼} A_{1} is an isomorphism \end{matrix}\} / (isomorphisms)

The Serre-Tate theory says that the deformation functor Def_{A_{1}} is represented by the formal scheme

\underline{Hom}_{ℤ_{p}} ( T_{p}(A_{1})(k) ⊗_{ℤ_{p}} T_{p}(A_{1}^{t})(k), 𝔾̂_{m} ) ,

or equivalently the formal scheme

( T_{p}(A_{1})(k) ⊗_{ℤ_{p}} T_{p}(A_{1}^{t})(k) )^{∨} ⊗_{ℤ_{p}} 𝔾̂_{m} .

Now suppose that (A_{1}, ι_{1}) is an ordinary abelian variety of dimension [F : ℚ] with multiplication by O_{F}. Then both T_{p}(A_{1})(k) and T_{p}(A_{1}^{t})(k) are free modules over O_{F} ⊗_{ℤ} ℤ_{p} of rank 1 by Lemma 8. Serre-Tate theory tells us that the formal completion of M_{F} at the point [(A_{1}, ι_{1})] ∈ M_{F} (k) is naturally isomorphic to

\underline{Hom}_{ℤ_{p}} ( T_{p}(A_{1})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} T_{p}(A_{1}^{t})(k), 𝔾̂_{m} )
≅ T_{p}(A_{1})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} T_{p}(A_{1}^{t})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} (D_{F}^{−1} ⊗_{ℤ} ℤ_{p}) ⊗_{ℤ_{p}} 𝔾̂_{m} .

The same statement is true for M_{F}^{Λ}, where (Λ, Λ_{+}) = (P(A_{1}, ι_{1}), P(A_{1}, ι_{1})_{+}), since M_{F}^{Λ} is étale over M_{F}. Assume in addition that (A_{1}, ι_{1}) is defined over \overline{𝔽_{p}}. Then the ring O_{(A_{1}, ι_{1})} := End_{O_{F}} (A_{1}, ι_{1}) is an order in a totally imaginary quadratic extension L of F, which is split at every place of F above p by Lemma 8. Its group of p-adic units U_{p} := (O_{(A_{1}, ι_{1})} ⊗_{ℤ} ℤ_{p})^{×} operates on the deformation space as follows: an element u ∈ U_{p} sends an element

q ( (A_{1}, ι_{1}); − , −) ∈ \underline{Hom}_{ℤ_{p}} ( T_{p}(A_{1})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} T_{p}(A_{1}^{t})(k), 𝔾̂_{m} )

to the element

T_{p}(A_{1})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} T_{p}(A_{1}^{t})(k) ∋ a ⊗ a^{t} ⟼ q ( (A_{1}, ι_{1}); u^{−1}(a), u^{t}(a^{t}) ) ,
∀ a ∈ T_{p}(A_{1})(k), ∀ a^{t} ∈ T_{p}(A_{1}^{t})(k) .

It is clear that the subgroup (O_{F} ⊗_{ℤ} ℤ_{p})^{×} of U_{p} operates trivially on the deformation space.

Now we resume the situation in the beginning of this section. In the rest of this section, we only consider equi-characteristic deformations, and all universal deformation functors and the formal tori representing them are understood to be over k. Let 𝔏^{×} (resp. 𝔉^{×}) be the algebraic torus over ℚ such that 𝔏^{×}(ℚ) = L^{×} (resp. 𝔉^{×}(ℚ) = F^{×}). The relative norm defines a homomorphism

N_{L/F} : 𝔏^{×} ⟶ 𝔉^{×} .

Let 𝔏_{0}^{×} = N_{L/F}^{−1} (𝔾_{m}) , an algebraic torus over ℚ. Clearly

𝔏_{0}^{×}(ℚ) = N_{L/F}^{−1} (ℚ^{×}) ⊆ L^{×} ,

and

𝔏_{0}^{×}(ℚ_{p}) = N_{L/F}^{−1} (ℚ^{×}) ⊆ (L ⊗_{ℚ} ℚ_{p})^{×} .

The group U_{p}^{0} = 𝔏_{0}^{×}(ℚ_{p}) ∩ U_{p} is a compact open subgroup of 𝔏_{0}^{×}(ℚ_{p}) . For any point x̃_{1} = [(A_{1}, ι_{1})] ∈ Z̃_{sm}^{or} defined over \overline{𝔽}_{p}, the stabilizer subgroup of x̃_{1} in the reduced prime-to-p F - Hecke correspondences contains those which come from 𝔏_{0}^{×}(ℚ) ∩ U_{p}. By the weak approximation theorem, 𝔏_{0}^{×}(ℚ) ∩ U_{p} is dense in U_{p}^{0} . Therefore the formal completion Z̃∧ ; x̃ of Z̃ at x̃ is stable under the action of U_{p}^{0} . (Had we used the reduced ℓ-power F -Hecke correspondences, we would have to replace 𝔏_{0}^{×}(ℚ) ∩ U_{p} by 𝔏_{0}^{×}(ℚ) ∩ (O_{(A_{1}, ι_{1})} ⊗_{ℤ} ℤ [\frac{1}{ℓ}])^{×} in the above, and run into the difficulty that the closure of 𝔏_{0}^{×}(ℚ)∩(O_{(A_{1}, ι_{1})} ⊗_{ℤ} ℤ [\frac{1}{ℓ}])^{×} may not be an open subgroup of U_{p}^{0} .) Consider the homomorphism

𝔏_{0}^{×} ⟶ 𝔏^{×}/𝔉^{×}

induced by the inclusion 𝔏_{0}^{×} → 𝔏^{×}. One checks easily that this homomorphism is faithfully flat. Its kernel is 1-dimensional but is not connected. Clearly the image of U_{p}^{0} in (𝔏^{×}/𝔉^{×})(ℚ_{p}) is an open subgroup of (𝔏^{×}/𝔉^{×})(ℚ_{p}) . Since L/F is split at every place of L above p,

(𝔏^{×}/𝔉^{×})(ℚ_{p}) = (L ⊗_{ℚ} ℚ_{p})^{×}/(F ⊗_{ℚ} ℚ_{p})^{×} .

Also there is a unique isomorphism

(L ⊗_{ℚ} ℚ_{p})^{×}/(F ⊗_{ℚ} ℚ_{p})^{×} ⟶^{∼} (F ⊗_{ℚ} ℚ_{p})^{×}

such that the composition

U_{p} ↪ 𝔏_{0}^{×}(ℚ_{p}) → (𝔏^{×}/𝔉^{×})(ℚ_{p}) = (L ⊗_{ℚ} ℚ_{p})^{×}/(F ⊗_{ℚ} ℚ_{p})^{×} ⟶^{∼} (F ⊗_{ℚ} ℚ_{p})^{×}

factorizes as

U_{p} ⟶^{χ} (O_{F} ⊗_{ℤ} ℤ_{p})^{×} ↪ (F ⊗_{ℚ} ℚ_{p})^{×} ,

and such that χ coincides with the character of the following O_{F} ⊗_{ℤ} ℤ_{p}-linear action of the group U_{p} on the free rank-one O_{F} ⊗_{ℤ}ℤ_{p}-module T_{p}(A_{1})(k)⊗_{( O_{F} ⊗_{ℤ}ℤ_{p})} T_{p}(A_{1}^{t})(k) :

U_{p} ∋ u ⟼ ( a ⊗ a^{t} ↦ u(a) ⊗ (u^{t})^{−1}(a^{t}) )
∀ a ⊗ a^{t} ∈ T_{p}(A_{1})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} T_{p}(A_{1}^{t})(k) .

From the above discussions and Lemma 8, we see that the formal subscheme Z̃∧ ; x̃ of

T_{p}(A_{1})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} T_{p}(A_{1}^{t})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} (D_{F}^{−1} ⊗_{ℤ} ℤ_{p}) ⊗_{ℤ_{p}} 𝔾̂_{m}

is stable under the action of an open subgroup of (O_{F} ⊗_{ℤ} ℤ_{p})^{×}. Here the action of (O_{F} ⊗_{ℤ} ℤ_{p})^{×} on

T_{p}(A_{1})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} T_{p}(A_{1}^{t})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} (D_{F}^{−1} ⊗_{ℤ} ℤ_{p}) ⊗_{ℤ_{p}} 𝔾̂_{m}

comes from the natural O_{F} ⊗_{ℤ} ℤ_{p}-module structure of

T_{p}(A_{1})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} T_{p}(A_{1}^{t})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} (D_{F}^{−1} ⊗_{ℤ} ℤ_{p}) .

Let

Y := T_{p}(A_{1})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} T_{p}(A_{1}^{t})(k) ⊗_{(O_{F} ⊗_{ℤ}ℤ_{p})} (D_{F}^{−1} ⊗_{ℤ} ℤ_{p}) ,

a free O_{F} ⊗_{ℤ} ℤ_{p}-module of rank 1. We have seen that

Z̃∧ ; x̃ ⊆ Y ⊗_{ℤ_{p}} 𝔾̂_{m}

is stable under the action of an open subgroup of (O_{F} ⊗_{ℤ} ℤ_{p})^{×} . Write

O_{F} ⊗_{ℤ} ℤ_{p} = O_{𝔭_{1}} × ⋯ × O_{𝔭_{r}} ,

where 𝔭_{1}, …, 𝔭_{r} are the prime ideals of O_{F} above p and O_{𝔭_{i}} is the maximal order in F_{𝔭_{i}} . Correspondingly, there is a decomposition

Y = Y_{1} ⊕ ⋯ ⊕ Y_{r} ,

where each Y_{i} is a free O_{𝔭_{i}}-module of rank 1: Y_{i} = O_{𝔭_{i}} · Y. We have seen that there exist open subgroups U_{i} ⊆ O_{𝔭_{i}}^{×} such that

Z̃∧ ; x̃ ⊆ (Y_{1} ⊗_{ℤ_{p}} 𝔾̂_{m}) × ⋯ × (Y_{r} ⊗_{ℤ_{p}} 𝔾̂_{m})

is stable under the action of U_{1} × ⋯ × U_{r}. We shall see that there are only a finite number of possibilities for Z̃∧ ; x̃:

Proposition 4. Let p be a prime number. Suppose that for each i = 1, …, r, O_{𝔭_{i}} is a complete discrete valuation ring of characteristic 0 with finite residue field κ_{i} ≅ 𝔽_{p^{n_{i}}} . Let Y_{i} be a free O_{𝔭_{i}}-module of rank 1 for i = 1, …, r. Suppose that

W ⊆ (Y_{1} ⊗_{ℤ_{p}} 𝔾̂_{m}) × ⋯ × (Y_{r} ⊗_{ℤ_{p}} 𝔾̂_{m})

is a smooth formal subscheme of (Y_{1} ⊗_{ℤ_{p}} 𝔾̂_{m}) × ⋯ × (Y_{r} ⊗_{ℤ_{p}} 𝔾̂_{m}) over an algebraically closed field k of characteristic p, which is invariant under the natural action of U_{1} × ⋯ × U_{r}, where each U_{i} is an open subgroup of O_{𝔭_{i}}^{×} , i = 1, …, r. Then there exists a subset ω ⊆ {1, …, r} such that

W = ∏_{i ∈ ω} Y_{i} ⊗_{ℤ_{p}} 𝔾̂_{m} ,

a product of formal subtori.

Proof. Let I be the ideal of W. Let R_{i} be the ring of regular formal functions of Y_{1} ⊗_{ℤ_{p}} 𝔾̂_{m}, i = 1, …, r. Let gr^{•}(R_{i}) be the graded ring of R_{i}. Then gr^{•}(R_{i}) is naturally isomorphic to the symmetric k-algebra S^{•} (X_{i} ⊗_{ℤ_{p}} k), where X_{i} = Hom_{ℤ_{p}}(Y_{i}, ℤ_{p}) , the dual of Y_{i}. The graded ring gr^{•}(R) of R = R_{1} ˆ⊗_{k} ⋯ ˆ⊗_{k}R_{r} is naturally isomorphic to gr^{•}(R_{1}) ⊗_{k} ⋯ ⊗_{k} gr^{•}(R_{r}) . The tangent cone of W is defined by the prime ideal gr^{•}(I) of gr^{•}(R) . It is generated by a k-vector subspace Ī_{W} of ⨁_{i=1}^{r} X_{i} ⊗_{ℤ_{p}} k ⊂ gr^{•}(R) because W is smooth.

There exists a natural number N ∈ ℕ such that 1 + p^{N} O_{𝔭_{i}} ⊆ U_{i} for each i = 1, …, r. For each i, let

Φ_{X_{i}, p^{n}} : X_{i} ⊗_{ℤ_{p}} 𝔽_{p} ⟶ S^{p^{n}} (X_{i} ⊗_{ℤ_{p}} 𝔽_{p})

be the map sending each element of X_{i} ⊗_{ℤ_{p}} 𝔽_{p} to its p^{n}-th power. Given any non-zero element α = ∑_{i=1}^{r} α_{i} in Ī_{W} with α_{i} ∈ X_{i} ⊗_{ℤ_{p}} 𝔽_{p} for i = 1, …, r. Suppose that α_{i_{0}} ≠ 0 , and pick an element f ∈ I with initial form α. Let ϵ_{i} : U_{i} → U_{1} ×⋯×U_{r} denote the injection of the i-th factor U_{j} into U_{1} ×⋯×U_{r}. Then a straightforward calculation shows that the initial form of ϵ_{i_{0}}(1 + p^{n}) · f − f is the image of α_{0} under

Φ_{X_{i_{0}} , p^{n}} ⊗_{𝔽_{p}} id_{k} : X_{i_{0}} ⊗_{ℤ_{p}} k = X_{i_{0}} ⊗_{ℤ_{p}} 𝔽_{p} ⊗_{𝔽_{p}} k
⟶ S^{p^{n}} (X_{i_{0}} ⊗_{ℤ_{p}} 𝔽_{p}) ⊗_{𝔽_{p}} k = S^{p^{n}} (X_{i_{0}} ⊗_{ℤ_{p}} k) .

Therefore Φ_{X_{i_{0}} , p^{n}} ⊗ id_{k}(α_{0}) ∈ gr^{•}(I) if n ≥ N . Since gr^{•}(I) is a prime ideal, it follows that the image of α_{i_{0}} under

id_{X_{i_{0}}} ⊗σ_{p}^{−n} : X_{i_{0}} ⊗_{ℤ_{p}} k ⟶ X_{i_{0}} ⊗_{ℤ_{p}} k

belongs to gr^{•}(I) for all n ≥ N, where σ_{p}^{−n} : k → k denotes the inverse of the n-th power of the Frobenius on k. Since this holds for every element α ∈ Ī_{W} , it follows that Ī_{W} is a product of subspaces defined over 𝔽_{p}: There exists 𝔽_{p}-subspaces Ī_{i} ⊆ X_{i} ⊗_{ℤ_{p}} 𝔽_{p}, i = 1, …, r, such that Ī_{W} = (Ī_{1} ⊕ ⋯ ⊕ Ī_{r}) ⊗_{𝔽_{p}} k.

Clearly each subspace Ī_{i} ⊆ X_{i} ⊗_{ℤ_{p}} 𝔽_{p} is a module under O_{𝔭_{i}} ⊗_{ℤ_{p}} 𝔽_{p}. We want to show that each Ī_{i} ⊆ X_{i} ⊗_{ℤ_{p}} 𝔽_{p} is either equal to the trivial subspace (0) or X_{i}⊗_{ℤ_{p}} 𝔽_{p} itself. Let π_{i} be a uniformiser of O_{𝔭_{i}} . Let e_{i} be the absolute ramification index for O_{𝔭_{i}} and write p = u_{i} · π^{e_{i}} with u_{i} ∈ O_{𝔭_{i}}^{×} . If Ī_{i} ≠ (0) , then it has to contain the subspace π_{i}^{e_{i}−1} · X_{i} ⊗_{ℤ_{p}} 𝔽_{p}. Therefore there exists an element f_{i} ∈ I with initial form π_{i}^{e_{i}−1} ·β_{i}, where β_{i} is the image in X_{i} ⊗_{ℤ_{p}} 𝔽_{p} of an O_{𝔭_{i}}-generator of X_{i}. Of course the initial form of the element ϵ_{i}(1+u_{i}π)·f_{i} − f_{i} is in gr^{•}(I) . A simple calculation shows that this initial form is equal to Φ_{X_{i}, p}(β_{i}) , the p-th power of the degree-one form β_{i}. Since gr^{•}(I) is a prime ideal, we get β_{i} ∈ gr^{•}(I) and hence β ∈ Ī_{i}. Thus if Ī_{i} ≠ (0) , then it contains a generator of X_{i} ⊗_{ℤ_{P}} 𝔽_{p}, hence is equal to X_{i} ⊗_{ℤ_{P}} 𝔽_{p}. We have shown that there exists a subset ω ⊆ {1, …, r} such that Ī_{W} = ⨁_{j∉ω} X_{j} ⊗_{ℤ_{P}} 𝔽_{p}.

We still have to show that W itself is a product of formal subtori. Pick a ℤ_{p}- basis for each X_{i}, we arrive at a ℤ_{p}-basis χ_{1}, …, χ_{a} for ⊕_{i∈ω} X_{i} and a ℤ_{p}-basis χ_{a+1}, …, χ_{a+b} for ⨁_{j∉ω} X_{j} . Then we have

R ≅ k[[x_{1}, …, x_{a}, y_{1}, …, y_{b}]] = k[[\underline{x}, \underline{y}]] ,

where x_{1} = χ_{1} −1 …, x_{a} = χ_{a} −1 , y_{1} = χ_{a+1} −1, …, y_{b} = χ_{a+b} −1 . Our result above says that there exist elements in I of the form

g_{j}(\underline{x}, \underline{y}) = y_{j} + Δ_{j}(\underline{x}, \underline{y}) j = 1, …, b ,

with Δ_{j}(\underline{x}, \underline{y}) ≡ 0 mod (\underline{x}, \underline{y})^{2} for each j = 1, …, b. They form a set of generators of the ideal I. An easy manipulation of power series shows that we can modify the g_{j}(\underline{x}, \underline{y})’s to get elements in I of the form

h_{j}(\underline{x}, \underline{y}) = y_{j} + ρ_{j}(\underline{x}) j = 1, …, b ,

such that ρ_{j}(\underline{x}) ∈ k[[\underline{x}]] and ρ_{j}(\underline{x}) ≡ 0 mod (\underline{x})^{2} for each j = 1, …, b. Again the h_{j}(\underline{x}, \underline{y})’s generate the ideal I. To conclude the proof of proposition 4, we only need to show that all the ρ_{j}(\underline{x})’s are equal to 0. To see this, consider the elements

η_{j}(\underline{x}) = (∏_{i∈ω} ϵ_{i}(1 + p^{N})) · h_{j}(\underline{x}, \underline{y}) − h_{j}(\underline{x}, \underline{y})

= (∏_{i∈ω} ϵ_{i}(1 + p^{N})) · ρ_{j}(\underline{x}) − ρ_{j}(\underline{x}) j = 1, …, b .

The initial form of η_{j}(\underline{x}) will be a non-zero form in k[\underline{x}] unless ρ_{j}(\underline{x}) = 0. However we know that gr^{•}(I) ⊆ k[\underline{x}, \underline{y}] is the ideal generated by y_{1}, …, y_{b}, which does not contain any non-zero form in k[\underline{x}] . Therefore all the ρ_{j}(\underline{x})’s are equal to 0. Hence the ideal I is generated by y_{1} …, y_{b}, i.e. W = ∏_{i ∈ ω} Y_{i} ⊗_{ℤ_{p}} 𝔾̂_{m} . Proposition 4 is proved. □

The statement in Proposition 4 can be globalized at the level of tangent spaces at the smooth points of Z̃. Recall that the tangent sheaf of M_{F} has a natural O_{F} -module structure. If at a point [(A_{x}, ι_{x})] of M_{F} (k) the Lie algebra Lie(A_{x}) is a free O_{F} ⊗_{ℤ} k-module of rank 1, then the stalk T_{M_{F} , x} of the tangent sheaf T_{M_{F}} of M_{F} at x is a free O_{F} ⊗_{ℤ} O_{M_{F} , x}-module of rank 1. So if {𝔭_{1}, …, 𝔭_{r}} are the primes of F above p as before, T_{M_{F} , x} of M_{F} decomposes as a direct sum:

T_{M_{F} , x} = ⨁_{i=1}^{r} T_{M_{F} , x}(𝔭_{i}) ,

and each T_{M_{F} , x}(𝔭_{i}) is a free O_{𝔭_{i}} ⊗_{ℤ_{p}} O_{M_{F} , x}-module of rank 1. Because of Lemma 5, M_{F} is smooth at ordinary points.

The following is an immediate consequence of Proposition 4 by faithfully flat descent.

Proposition 5. For each irreducible component W of the smooth locus Z̃_{sm} of Z̃ there exists a subset ω ⊆ {1, …, r} such that the tangent sheaf T_{W} ⊆ T_{M_{F}} ⊗_{O_{M_{F}}} O_{W} of W is equal to

⨁_{i∈ω} T_{M_{F}} (𝔭_{i}) ⊗_{O_{M_{F}}} O_{W} .

Remark. In fact one can deduce a stronger consequence of Proposition 4 in the situation of Proposition 5: For each irreducible component W of Z̃_{sm}^{or} there exists a subset ω ⊆ {1, …, r} such that in the direct sum decomposition of the p-divisible group A_{W} [p^{∞}] of the universal abelian scheme A_{W} with multiplication by O_{F}

A_{W} [p^{∞}] = ⨁_{i∈ω} A_{W} [𝔭_{i}^{∞}] ⊕ ⨁_{j∉ω} A_{W} [𝔭_{j}^{∞}] ,

the first factor ⨁_{i∈ω} A_{W} [𝔭_{i}^{∞}] is universal at every point of W (k), while the second factor ⨁_{j∉ω} A_{W} [𝔭_{j}^{∞}] is the direct sum of a multiplicative p-divisible group with an étale p-divisible group. Hence the statement in Proposition 4 holds for every point of Z̃_{sm}^{or}(k) . Since we will not need this stronger statement, the proof is omitted.

5. Inspection at the supersingular points

Proposition 6. Let F be a totally real number field, Λ be a projective O_{F} -module of rank 1 with a notion of positivity. Let Z̃ be a Zariski closed subscheme of M_{F}^{Λ} over an algebraically closed field k of characteristic p > 0 , which is stable under all reduced ℓ-power F -Hecke correspondences for a prime number ℓ ≠ p. Then Z̃ contains supersingular points, and is non-proper only if Z̃ is equal to M_{F}^{Λ} .

Proof. The idea of the proof is already sketched in §1. Let Z̃^{∗} be the closure of Z̃ in the minimal compactification M_{F}^{Λ ∗} of M_{F}^{Λ} . The minimal compactification of Hilbert-Blumenthal moduli spaces can be constructed using the methods of [FC] chap. 5; it is discussed in [Ch2]. The boundary of M_{F}^{Λ ∗} consists of isolated points usually referred to as cusps. Clearly Z̃ is stable under all reduced prime-to-p F - Hecke correspondences. Because M_{F}^{Λ ∗} is proper over Spec k, the subscheme Z̃ of M_{F}^{Λ} is proper over Spec k if and only if its closure Z̃^{∗} in M_{F}^{Λ ∗} is equal to Z̃ itself. In other words, Z̃ is non-proper over Spec k if and only if Z̃^{∗} contains a cusp. If Z̃^{∗} contains a cusp, then by a calculation similar to and simpler than that of Proposition 2, Z̃ is equal to M_{F}^{Λ} , since M_{F}^{Λ} is irreducible and geometrically normal. Of course then Z̃ contains a supersingular point.

Assume now that Z̃ is proper over k. We can apply the main result of [EO], since Λ contains separable polarizations, and all results in [EO] remain true for separable polarizations and not just principal polarizations: the same proof works. The main result of [EO] implies that there is a natural stratification of M_{F}^{Λ} which comes from the p-torsion points A[p] of the universal abelian scheme A over M_{F}^{Λ} , and each stratum is quasi-affine. Therefore unless Z̃ is 0-dimensional, it cannot be contained in the generic stratum. So Z̃ meets some stratum S_{α} of lower type than the generic stratum if dim(Z̃) > 0. Again Z̃ ∩ S_{α} is stable under all reduced ℓpower F -Hecke correspondences. If this intersection is not 0-dimensional, its closure meets some stratum S_{β} of lower type than S_{α}. So the intersection Z̃ ∩ S_{β} is non-empty. This argument can be continued, eventually we reach some stratum S_{γ} such that Z̃ ∩ S_{γ} is non-empty and 0-dimensional. Therefore Z̃ contains a point with finite reduced ℓ-power F -Hecke orbit. By Lemma 7, this points has to be supersingular. This proves Proposition 6. □

Proposition 7. Let k be an algebraically closed field of characteristic p > 0 . Let F be a totally real number field and let Λ be a projective O_{F} -module of rank 1, with a notion of positivity. If Z̃ is the Zariski closure of the reduced prime-to-p F -Hecke orbit of an ordinary point of M_{F}^{Λ}(k) , then Z̃ = M_{F}^{Λ} .

Proof. Clearly Z̃ is reduced, and the smooth ordinary locus Z̃_{sm}^{or} is open and dense in Z̃. Assume that Z̃ is not equal to M_{F}^{Λ} . We want to get a contradiction.

By Proposition 6, Z̃ contains a supersingular point x_{s} = [(A_{s}, ι_{s}, h_{s}] ∈ M_{F}^{Λ}(k) . As before let {𝔭_{1}, …, 𝔭_{r}} be the prime ideals of O_{F} above p. The p-divisible group A_{s}[p^{∞}] of A_{s} decomposes into a direct sum

A_{s}[p^{∞}] = A_{s}[𝔭_{1}^{∞}] ⊕ ⋯ ⊕ A_{s}[𝔭_{r}^{∞}] ,

corresponding to the direct sum decomposition O_{F} ⊗_{ℤ} ℤ_{p} = O_{𝔭_{1}} ⊕ ⋯ ⊕ O_{𝔭_{r}} . Each A_{s}[𝔭_{i}^{∞}] is a p-divisible group with an action by O_{𝔭_{i}} , and dim(A_{1}[𝔭_{i}^{∞}]) = 2 [F_{𝔭_{i}} : ℚ_{p}] . By the Serre-Tate theorem, the formal completion of M_{F}^{Λ} at the point x_{s} is isomorphic to the universal deformation space of (A_{s}[p^{∞}], ι_{s}) over k. In turn, this universal deformation space decomposes into the product of the universal deformation spaces of (A_{s}[𝔭_{i}^{∞}], ι_{i}) over k, where i ranges from 1 to r, and ι_{i} denotes the action of O_{𝔭_{1}} on A_{s}[𝔭_{i}^{∞}] . Let 𝔇_{i} denote the universal deformation spaces of (A_{s}[𝔭_{i}^{∞}], ι_{i}) over k. The formal completion Z̃_{x_{s}}^{∧} of Z̃ at x_{s} is a closed formal subscheme of 𝔇 = 𝔇_{1} × ⋯ × 𝔇_{r}.

Consider End(A_{s}, ι_{s}) and End^{0}(A_{s}, ι_{s}) = End_{O_{F}} (A_{s}, ι_{s}) ⊗_{O_{F}} F . By Lemma 6 iii, B = End^{0}(A_{s}, ι_{s}) is a totally definite quaternion algebra over F which is unramified at all places of F which are prime to p, and for places 𝔭 of F above p, B is ramified at 𝔭 if [F_{𝔭} : ℚ_{p}] is odd, unramified if [F_{𝔭} : ℚ_{p}] is even. When tensored with ℚ_{p} the quaternion algebra B decomposes into a direct sum

B ⊗_{ℚ} ℚ_{p} = (B ⊗_{F} F_{𝔭_{1}}) ⊕ ⋯ ⊕ (B ⊗_{F} F_{𝔭_{r}}) ,

each B ⊗_{F} F_{𝔭_{1}} is a quaternion algebra over F_{𝔭_{i}} . Correspondingly we have a decomposition of the order End(A_{s}, ι_{s}) ⊗_{ℤ} ℤ_{p} in B ⊗_{ℚ} ℚ_{p}:

End(A_{s}, ι_{s}) ⊗_{ℤ} ℤ_{p} = (End(A_{s}, ι_{s}) ⊗_{O_{F}} O_{𝔭_{1}}) ⊕ ⋯ ⊕ (End(A_{s}, ι_{s}) ⊗_{O_{F}} O_{𝔭_{r}})
= (End(A_{s}[𝔭_{1}^{∞}], ι_{1}) ⊕ ⋯ ⊕ (End(A_{s}[𝔭_{r}^{∞}], ι_{r})

Let B_{1}^{×} be the group of elements of B with reduced norm 1. Similarly let (B ⊗_{ℚ} ℚ_{p})_{1}^{×} (resp. (B ⊗_{F} F_{𝔭_{i}})_{1}^{×}) be the group of elements of (B ⊗_{ℚ} ℚ_{p})^{×} (resp. (B ⊗_{F} F_{𝔭_{i}})^{×}) with reduced norm 1. Let B_{red}^{×} be the group of elements of B^{×} whose reduced norms are in ℚ^{×}. Let U_{p} = (End(A_{s}, ι_{s})⊗_{ℤ} ℤ_{p})^{×}, U_{𝔭_{i}} = (End(A_{s}, ι_{s})⊗_{O_{F}} O_{𝔭_{1}})^{×} = (End(A_{s}[𝔭_{i}^{∞}], ι_{i})^{×}, i = 1, …, r. Then U_{p} = U_{𝔭_{1}} ×⋯×U_{𝔭_{r}} . Each U_{𝔭_{i}} operates on 𝔇_{i} ‘by changing the marking of the closed fiber’. Similarly the group U_{p} acts on 𝔇, and is just the product of the actions of U_{𝔭_{i}} on 𝔇_{i}, i = 1, …, r. Let U_{p}^{1} = U_{p} ∩ (B ⊗_{ℚ} ℚ_{p})_{1}^{×} and let U_{𝔭_{i}}^{1} = U_{𝔭_{k}} ∩ (B ⊗_{F} f_{𝔭_{i}})_{1}^{×}. Clearly

U_{p}^{1} = U_{𝔭_{1}}^{1} × ⋯ × U_{𝔭_{r}}^{1} .

The stabilizer subgroup for (A_{s}, ι_{s}) in the reduced prime-to-p F -Hecke correspondences contains those which come from B_{red}^{×} ∩ U_{p}. Since B_{1}^{×} ∩ U_{p}^{1} is dense in U_{p}^{1} by the weak approximation theorem, the closed subscheme Z̃_{x_{s}}^{∧} ⊆ 𝔇 is stable under the action of U_{p}^{1}.

The supersingular point x_{s} belongs to the closure of an irreducible component W of Z̃_{sm}^{or}. By Proposition 5, there exists a subset ω ⊆ {1, …, r} such that the tangent sheaf T_{W} ⊆ T_{M_{F}} ⊗_{O_{M_{F}}} O_{W} of W is equal to

⨁_{i∈ω} T_{M_{F}} (𝔭_{i}) ⊗_{O_{M_{F}}} O_{W} .

Since we assumed that Z̃ is not equal to M_{F}^{Λ} , ω ≠ {1, …, r}.

Write 𝔇_{i} = Spf(R_{i}) , i = 1, …, r, 𝔇 = Spf(R_{1} ˆ⊗_{k} ⋯ ˆ⊗_{k}R_{r}) , and Z̃_{x_{s}}^{∧} = Spf(R/I) ⊂ Spf(R) = 𝔇. There exists a formal curve ξ : Spf(k[[t]]) → Z̃_{x_{s}}^{∧} which is generically ordinary. In other words the universal abelian variety over R, when base changed to k((t)) via R → k[[t]] ↪ k((t)) , gives an ordinary abelian variety over k((t)) . Equivalently, if we write ξ(t) = ( ξ_{1}(t), …, ξ_{r}(t) ) , where ξ_{i} : Spf(k[[t]]) → 𝔇_{i} is the i-th component of ξ, then all x_{1}(t), …, ξ_{r}(t) are generically ordinary. Moreover, we may assume that ξ gives a map from Spec k((t)) to Z̃_{sm}^{or}. In other words the jacobian criterion for (R, I) is satisfied for the homomorphism R → k((t)) and ξ is generically ordinary.

Pick any element j ∈ {1, …, r}, j ∉ ω. For any element u_{j} ∈ U_{𝔭_{i}}^{1} ⊆ U_{p}, u_{j} ·ξ is again a formal curve in Z. For any i ≠ j, the i-th component of u_{j} ·ξ is equal to ξ_{i}. If u_{j} is close to the identity, then the j-th component of u_{j} · ξ is close to ξ_{j} in the sense that these two homomorphisms from R_{j} to k[[t]] are congruent modulo high powers of t. On the other hand for u_{j} outside of a subalgebra of B ⊗_{F} F_{𝔭_{i}} of dimension at most 2 over F_{𝔭_{i}} , the j-th component (u_{j} · ξ)_{j} is not equal to ξ_{j} : Since ξ is generically ordinary, the endomorphism algebra of the p-divisible group with O_{𝔭_{j}} -multiplication (A_{s}[𝔭_{j}^{∞}], ι_{j})⊗_{R_{j}, ξ_{j}} k((t)) has rank at most 2 over O_{𝔭_{j}} . So there exist elements u_{j} ∈ U_{𝔭_{i}}^{1} such that (u_{j} ·ξ)_{j} is not equal to ξ_{j} , while congruent to ξ_{j} to arbitrarily high power of t. This implies that at the k((t)) -valued point R → k((t)) of W the tangent space of W contains some element with non-zero 𝔭_{j}-component. This can be checked using the Taylor expansion for instance. By Proposition 5, we must have j ∈ ω. This is a contradiction. Proposition 7 is proved. □

Remark. In the proof of Proposition 7 we have used very little information on the action of the automorphism group U_{p} of the closed fiber on the deformation space 𝔇 at a supersingular point x_{s}. Ideally we should be able to get much more results on this action, and prove an analogue of question (Q 2) in this setting.

Theorem 1. Let k be an algebraically closed field of characteristic p > 0 . Let F be a totally real number field and let Λ be a projective O_{F} -module of rank 1, with a notion of positivity. If Z̃_{ℓ} is the Zariski closure of the reduced ℓ-power F -Hecke orbit of an ordinary point of M_{F}^{Λ}(k) , then Z̃_{ℓ} = M_{F}^{Λ} .

Proof. First we show that Z̃_{ℓ} ⊆ M_{F}^{Λ} is stable under all reduced prime-to-p Hecke correspondences. By Proposition 6, Z̃_{ℓ} contains a supersingular point x_{s} = [(A_{s}, ι_{s}, h_{s})] in M_{F}^{Λ}(k) . We shall use the notation in the proof of Proposition 7. Especially B = End^{0}(A_{s}, ι_{s}) is a totally definite quaternion algebra over F . Let O_{B} = End(A_{s}, ι_{s}) , an order in B. Let G_{red} be the algebraic group over ℚ whose ℚ-rational points consists of elements in B^{×} with reduced norm in ℚ^{×}. For any prime number ℓ′, G_{red}(ℚ_{ℓ′}) consist of elements of (B ⊗_{ℚ} ℚ_{ℓ′})^{×} with reduced norm in ℚ_{ℓ′}^{×}. Let K_{ℓ′} = G_{red}(ℚ_{ℓ′}) ∩ (O_{B} ⊗_{ℤ} ℤ_{ℓ′})^{×}, a compact open subgroup of G_{red}(ℚ_{ℓ′}) . The derived group G_{red}^{der} of G_{red} is simply connected and ℚ-simple. Its group of real points G_{red}^{der}(ℝ) is compact, while for any prime number ℓ′ ≠ p the group G_{red}^{der}(ℚ_{ℓ′}) is noncompact. The quotient G_{red}/G_{red}^{der} is isomorphic to 𝔾_{m} via the reduced norm map. By the strong approximation theorem, G_{red}(ℚ) · G_{red}(ℚ_{ℓ}) is dense in G_{red}(𝔸_{ℚ}) . Especially G_{red}(ℚ) ∩ ∏_{ℓ′≠ℓ} K_{ℓ′} is dense in K_{p}. Since G_{red}(ℚ) ∩ ∏_{ℓ≠ℓ′} K_{ℓ′} is contained in the stabilizer subgroup of x_{s}, the formal completion Z_{ℓ,x_{s}}^{∧} of Z_{ℓ} at x_{s} is stable under the action of K_{p}. The strong approximation theorem also implies that the reduced prime-to-p F -Hecke orbit of x_{s} is equal to the reduced ℓ-power F -Hecke orbit of x_{s}, since G_{red}(ℚ) · G_{red}(ℚ_{ℓ}) is dense in G_{red}(𝔸_{f,ℚ}) . Therefore for any reduced prime-to-p F -Hecke correspondence γ, the union Z̃_{ℓ} ∪γ(Z̃_{ℓ}) of Z̃_{ℓ} with the image of Z̃_{ℓ} under γ coincides with Z̃_{ℓ} at the formal completion of the supersingular points of Z̃_{ℓ} ∪ γ(Z̃_{ℓ}) . Clearly Z̃_{ℓ} ∪γ(Z̃_{ℓ}) is stable under all reduced prime-to-ℓ F -Hecke correspondences of M_{F}^{Λ} . So Z̃_{ℓ} ∪ γ(Z̃_{ℓ}) is the union of Z̃_{ℓ} with a closed subscheme W_{γ} of M_{F}^{Λ} , such that W_{γ} does not contain supersingular points and is stable under all reduced ℓ-power F -Hecke correspondences on M_{F}^{Λ} . Hence W_{ℓ} is empty by Proposition 6. This shows that Z̃_{ℓ} is stable under all reduced prime-to-p F -Hecke correspondences on M_{F}^{Λ} . Therefore Z̃_{ℓ} = M_{F}^{Λ} by Proposition 7. Theorem 1 is proved. □

Clearly Proposition 3 and Theorem 1 imply

Theorem 2. Let d_{1}, …, d_{g} be positive integers such that d_{1} | ⋯ | d_{g}, and let δ = (d_{1}, …, d_{g}) . Let k be an algebraically closed field of characteristic p > 0 .

Then for any prime ℓ ≠ p, the ℓ-power Hecke orbit of any ordinary point of A_{g,δ} is Zariski dense in A_{g,δ}. In particular, the ℓ-power Hecke orbit of any ordinary point of A_{g} is Zariski dense in A_{g}.

References

[AMRT] A. Ash, D. Mumford, M. Rapoport and Y. Tai, Smooth Compactification of Locally Symmetric Spaces, Math. Sci. Press, 1975.

[Bo] A. Borel, Properties and linear representations of Chevalley groups, Seminar on algebraic groups and related finite groups, Lecture Notes in Math. 131, 1970, pp. 1-55.

[Ch1] C.-L. Chai, Compactification of Siegel Moduli Schemes, Lecture Notes Series 107, London Math. Soc., London, 1985.

[Ch2] , Arithmetic minimal compactification of Hilbert-Blumenthal moduli spaces, Appendix to Andrew Wiles “The Iwasawa conjecture for totally real fields, Annals of Math. 131 (1990), 541-554.

[Ch3] , The group action on the closed fiber of the Lubin-Tate moduli space, preprint (1994).

[De] P. Deligne, Variétés de Shimura: Interprétation modulair, et techniques de construction de modèles canoniques, Automorphic Forms, Representations, and L-functions (A. Borel and W. Casselman, eds.), Proc. Symp. Pure Math. 33, part 2, AMS, 1979, pp. 247–290.

[DP] P. Deligne and G. Pappas, Singularités des espaces de modules de Hilbert, en les caractéristiques divisant le discriminant, Compos. Math. 90 (1994), 59–79.

[Ek1] T. Ekedahl, On supersingular curves and abelian varieties, Math. Scand 60 (1987), 151-178.

[Ek2] T. Ededahl, The action of monodromy on torsion points of jacobians, Arithmetic Algebraic Geometry, Texel 1989, Eds. G. van der Geer, F. Oort, J. Steenbrink, Progress in Math. 89, Birkhäuser, 1991, pp. 41-49.

[EO] T. Ekedahl and F. Oort, Connected subsets of a moduli space of abelian varieties, preprint (1994).

[F] G. Faltings, Arithmetische Kompaktifizierung des Modulraums der abelschen Varietäten, Lecture Notes in Math. 1111, Springer-Verlag, 1985, pp. 321-383.

[FC] G. Faltings and C.-L. Chai, Degeneration of Abelian Varieties, Ergebnisse der Mathematik und ihrer Grenzgebiete, 3. Folge, Band 22, Springer-Verlag, 1990.

[Gr] B. Gross, On canonical and quasi-canonical liftings, Inven. Math. 84 (1986), 321-326.

[GH] B. H. Gross and M. J. Hopkins, Equivariant vector bundles on the Lubin-Tate moduli space (to appear).

[G] A. Grothendieck, Groupes de Barsotti-Tate et Cristaux de Dieudonné, Les Presses de l’Université de Montréal, 1974.

[dJ] A. J. de Jong, The moduli space of polarized abelian varieties, Math. Ann. 295 (1993), 485-503.

[Ka1] N. M. Katz, Travaux de Dwork, Séminaire Bourbaki 1971/72, exposé 409, Lecture Notes in Math. 317, Springer-Verlag, 1973, pp. 69-190.

[Ka2] , Serre-Tate local moduli, Séminaire de Géométrie Algébrique d’Orsay 1976-78, Expposé Vbis, Surface Algébriques, Lecture Notes in Math. 868, Springer-Verlag, 1981, pp. 138-202.

[Ko] N. Koblitz, P-adic variation of the zeta-function over families of varieties defined over finite fields, Compos. Math. 31 (1975), 119–218.

[LR] R. Langlands and M. Rapoport, Shimuravarietäten und Gerben, J. reine angew. Math. 378 (1987), 113-220.

[LT] J. Lubin and J. Tate, Formal moduli for one-parameter formal Lie groups, Bull. Soc. Math. France 94 (1966), 49-60.

[Me] W. Messing, The Crystals Associated to Barsotti-Tate Groups: with Applications to Abelian Schemes, Lecture Notes in Math. 370, Springer-Verlag, 1972.

[Mi1] J. S. Milne, The points on a Shimura variety modulo a prime of good reduction, The Zeta Function of Picard Modular Surfaces (R. Langlands and D. Ramakrishnan, eds.), Les Publications CRM, Montréal, 1992, pp. 153-255.

[Mi2] , Shimura varieties and motives, Proc. Symp. Pure Math. (1994). [M-B] L. Moret-Bailly, Pinceaux de Variétés Abéliennes, Astérisque 129, Soc. Math. France,

1985.

[Mu] D. Mumford, Abelian Varieties, Tata Inst., Studies in Math. 5, Oxford University Press, 1974.

[N] P. Norman, An algorithm for computing local moduli of abelian varieties, Ann. Math. 101 (1975), 499-509.

[NO] P. Norman and F. Oort, Moduli of abelian varieties, Annals of Math. 112 (1980), 413-439.

[OO] T. Oda and F. Oort, Supersingular abelian varieties, Proc., Kyoto Univ., Kyoto, 1977, Kinokuniya Book Store, Tokyo, 1978, pp. 595-621.

[Oort] F. Oort, The isogeny class of a CM-type abelian variety is defined over a finite extension of the prime field, J. Pure Appl. Algebra 3 (1973), 399-408.

[Ra] M. Rapoport, Compactifications de l’espace de modules de Hilbert-Blumenthal, Compo. Math. 36 (1978), 255-335.

[Sh] G. Shimura, On analytic families of polarized abelian varieties and automorphic functions, Ann. of Math. 78 (1963), 149-192.

[Ta] J. Tate, Classes d’isogeny de variétés abéliennes sur un corps fini (d’apès T. Honda), Sém. Bourbaki Exp. 352 (1968/69), Lecture Notes in Math. 179, Springer Verlag, 1971.

[Wa] S. P. Wang, On density properties of S-subgroups of locally compact groups, Annals of Math. 94 (1971), 325-329.

Department of Mathematics, University of Pennsylvania, Philadelphia, PA 19104-6395, U.S.A.

E-mail address: chai@math.upenn.edu